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EXECUTIVE  SUMMARY 


RESEARCH  ON  MINORITY/DISADVANTAGED 
SUPPORT  PROGRAMS 


Over  the  past  twenty  years  of  its  existence,  the  Health  Careers  Opportunity  Program 
(HCOP)  has  facilitated  the  entrance  and  graduation  of  thousands  of 
minority /disadvantaged  students  into  health  professions  schools.  Between  1990  and 
1994,  an  average  of  8,600  students  (nation-wide)  participated  in  HCOP  each  year. 
Approximately  20  percent  of  these  students  attended  post  baccalaureate  programs.  Since 
HCOP's  initiation,  on  an  average,  70  percent  of  its  program  participants  who  applied  to 
health  professions  schools  have  been  successful  in  gaining  admission.1 
As  the  result  of  shifting  political  and  economic  priorities,  new  issues,  barriers  and 
limitations  have  arisen  for  HCOP-funded  programs.  Factors  such  as  health  care  reform 
are  causing  a  ripple  effect  throughout  the  health  care  system.  As  hospitals  and  health 
agencies  scale  back  their  operations,  fewer  slots  are  available  for  residencies  and  entry 
level  health  professional  positions.  In  response  to  this  reality  and  in  conjunction  with 
shifting  perceptions  of  the  need  for  affirmative  action  programs  within  the  current 
market  environment,  prospective  minority /disadvantaged  health  professions  school 
applicants  are  facing  increasing  competition  for  a  decreasing  number  of  slots.  As  a 
result,  indications  point  to  the  fact  that  while  the  number  of  minority /disadvantaged 
students  applying  to,  and  graduating.from,  health  professions  schools  has  significantly 
increased2,  the  ratio  of  applicants  accepted  and  successfully  graduating,  relative  to  the 
total  number  of  applicants,  has  actually  decreased3.  These  are  some  of  the  key  issues 
and  challenges  facing  HCOP  and  prospective  minority /disadvantaged  health  professions 
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students.   This  report  has  been  prepared  to: 

•  identify  HCOP  participant  selection  variables  that  can  be  useful  in  matching 
minority/disadvantaged  students  with  the  HCOP  approaches  most  likely  to  help 
them  to  become  more  competitive,  especially  in  light  of  increasingly  more  difficult 
health  professions  school  acceptance  criteria; 

•  highlight  effective  programming  and  educational  interventions  that  have  resulted 
in  statistically  improved  course  grades  and  higher  levels  of  retention  of  at-risk 
minority/disadvantaged  students  within  the  health  professions  schools;  and, 

•  research  the  barriers  which  discourage  minority/disadvantaged  students  from 
participating  in  the  sciences  (see  Vol  II  Report,  Literature  Search  on 
Minority/Disadvantaged  Support  Programs). 

Our  goal  is  to  provide  HCOP  managers  with  strategies  and  tools  that  can  be  useful  in 
designing  future  interventions  based  on  proven  approaches  that  will  positively  effect  the 
number  of  minority/disadvantaged  students  each  year,  who  are  accepted  into  health 
professions  school  and  graduate  in  a  timely  manner.  Additionally,  this  research  has  led 
to  the  identification  of  tracking  and  valuative  processes  to  maximize  the  impact  of 
limited  HCOP  resources.  The  analyses  presented  in  this  report  reflect  information 
gained  through  an  expansive  literature  search,  thorough  examination  of  progress  reports 
provided  by  the  Division  of  Disadvantaged  Assistance  (DDA)  and  the  recommendations 
of  the  HCOP  Work  Group  (HWG)  [see  appendix  1]  which  was  established  in  support 
of  this  research  effort. 


CHAPTER  I,  Introduction  encompasses  background  information  on  the  issues 
facing  minority/disadvantaged  representation  in  the  health  professions.  This  chapter 
details  the  role  of  federal  support  programs,  such  as  HCOP,  in  increasing  these  numbers. 
Also  detailed  in  this  section  are  the  'questions  addressed'  (and  objectives)  through  this 
research  and  the  methodology  used  to  address  them.  The  objectives  of  this  analysis,  as 
defined  in  the  "Scope  of  Work,"  involve:  1)  characterize  those  factors  which  dissuade 
minority/disadvantaged  students  from  pursuing  science  programs  and  contribute  to 
high  drop  out  rates4,  2)  based  on  the  review  of  current  and  previous  HCOP 
interventions,  identify  effective  learning  strategies  which  can  be  replicated /adapted  at 
other  locations;  and,  3)  develop  selection  criteria /indicators  which  can  be  utilized  by 
HCOP  projects  to  select  those  students  most  likely  to  succeed  as  the  result  of  federal 
support  programs  like  HCOP.  The  literature  review  which  was  completed  prior  to  the 
development  of  this  report,  provides  a  preamble  for  the  initiation  of  HCOP  and  HCOP- 
type  interventions.  For  example:  the  Association  of  American  Medical  Colleges'  Project 
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"3000  by  2000,"5  calls  for  the  development  of  several  national  networks  and  databases 
to  track  programs  (as  well  as  individual  students)  that  can  help  identify  a  critical  pool 
of  minority /disadvantaged  candidates  for  health  professions  training.  To  complement 
this  and  other  efforts,  the  National  Science  Foundation's  Urban  Systemic  Initiative  is 
focused  on  producing  dramatic  reform  in  the  delivery  of  math  and  science  instruction 
at  the  secondary  school  level,  which  should  supply  the  health  professions  "pipeline"  with 
increasing  numbers  of  well-prepared  minority  high  school  graduates. 

These  approaches,  however,  do  not  necessarily  respond  to  the  immediate  and 
intermediate  need  for  more  minority /disadvantaged  health  profession  school  students. 
The  research  findings  support  the  need  to  continue  the  supplemental  learning  and 
academic  enrichment  efforts  funded  by  the  Health  Careers  Opportunity  Program 
(HCOP).  The  information  presented  within  this  report  indicates  that  many  of  these 
programs  have  provided  the  critical  element  of  assistance  needed  to  facilitate  the 
increased  success  of  minority /disadvantaged  students  in  entering  into  and  graduating 
from,  health  professions  schools  in  a  timely  manner. 

The  subsequent  chapters  of  this  report  will  present  a  profile  of  selective  HCOP  project 
activities,  present  insights  into  their  effectiveness  and  opportunities  for  enhancement  of 
the  over-all  program.  Each  chapter  concludes  with  a  list  of  recommendations  which 
correspond  to  proposed  guidelines.   The  following  chapters  of  this  report  will  detail: 

•  Chapter    II,    Effective    Learning    Approaches:    'Improving    Cognitive    Skills'    in 
Minority /Disadvantaged  Student  Populations 

•  Chapter  III,  Selection  of  the  HCOP  Participants 

•  Chapter  IV,  Assessment  of  HCOP  Project  Data 

•  Chapter  V,  A  Sampling  of  Interventions  Designed  to  Improve  the  Competitiveness 
of  Minority /Disadvantaged  Students 

•  Chapter  VI,  Comprehensive  Listing  of  Recommendations,  Related  Guidelines  and 
Rationales 
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KEY  RECOMMENDATIONS 

Based  upon  the  research  and  deliberations  by  the  HCOP  Work  Group,  the  following 
RECOMMENDATIONS  could  provide  a  foundation  for  HCOP  to  build  upon  its 
success  in  significantly  increasing  the  numbers  of  underrepresented  minorities 
providing  health  services. 

#1  Encourage  all  HCOP  Grantees  to  incorporate  HCOP  in  their 
program  name/title,  in  a  manner  which  is  appropriate  within 
the  context  of  their  individual  projects.  At  the  same  time, 
empower  HCOP  student  participants  as  partners  in  the 
program  by  formally  designating  them  as  MHCOP  Fellows." 
There  is  an  urgent  need  for  Grantees  to  develop  practices  that  better 
enables  students,  advisors  and  resource  allocators  to  know  that  HCOP 
provides  funding  and  technical  support  for  the  program.  The 
empowerment  concept  is  needed  both  to  increase  opportunities  for  HCOP 
participant  (student-specific)  level  tracking  throughout  their  health 
professions  educational  development  and  to  expand  options  for  measuring 
outcomes.  It  is  additionally  needed  to  enhance  participant  self-perception 
of  HCOP  students  and  to  entrust  them  with  responsibility  of  continuing 
their  association  with  HCOP  once  they  have  completed  the  program. 

#2  Encourage  HCOP  Grantees  to  utilize  both  cognitive  and  non- 
cognitive  variables  as  a  basis  for  the  selection  of  HCOP 
participants  and  the  matching  of  those  participants  to  the 
programs  which  are  most  likely  to  benefit  them.  This  is  needed  to 
provide  a  more  representative  assessment  of  each  applicant's  relative  academic 
risk  and  to  ensure  that  participants  are  matched  with  the  programs  that  are 
most  likely  to  benefit  them. 

#3  PDA  should  develop"  a  "universal  minimum  data  set"  (as 
discussed  in  Chapter  IV)  applicable  to  most  college  and  health 
professions  school  HCOP  programs  which  includes  both 
cognitive  and  non-  cognitive  variables.  PDA  should 
additionally  encourage  programs  to  include  questions  in  all 
participant  applications  which  will  elicit  the  necessary  data.  This 
is  needed  to  ensure  the  availability  of  reliable  comparable  data  which  can  be 
used  to  accurately  assess  HCOP's  impact  both  locally  and  nationwide,  and,  to 
identify  truly  effective  models. 
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#4  Encourage  HCOP  Grantees  to  become  familiar  with 
collaborative  learning  approaches,  to  use  them  in  a  way  that 
can  best  be  supported  within  their  existing  resources,  and,  are 
most  appropriate  in  meeting  the  needs  of  their  participants.  This 
is  needed  in  light  of  the  research  and  analysis  presented  in  this  report  which 
indicates  that  collaborative  learning  is  effective  in  improving  cognitive  skills 
and  non-cognitive  attributes. 

#5  Support  the  establishment  of  a  "National  Student  Academic 
Enrichment  Match  Service  Center,"  within  the  private  sector, 
which  will  attempt  to  match  a  student's  aspirations  (medical 
school,  dental  school  etc),  academic  profile,  existing  financial 
circumstance,  grade  level  and  geographic  location,  with  the 
curriculum  and  support  service  programs  or  courses  which  best 
address  his/her  needs.  The  activity  recommended  would  not  require  a 
complicated  operational  program  in  order  to  provide  assistance  to  students  and 
their  advisors.  This  service  is  urgently  needed  since  students  are  not  in  a 
position  to  effect  such  matches  on  their  own,  and  currently,  the  available 
information  on  programs  does  not  allow  students  to  identify  programs  that 
hold  the  most  promise  of  responding  to  his/her  needs. 

#6  Establish  a  Standing  Guideline  and  Diagnostic/Intervention 
Review  Committee  which  includes,  to  the  extent  possible:  a) 
HCOP  Grantee  staff  b)  recognized,  technical  experts  in  student 
testing,  c)  known  authorities  on  cognitive  learning  strategies  and 
d)  program  evaluation  specialists.  The  Committee  could  provide  a 
source  of  technical  assistance  consultants  which  the  programs  and  their 
institutions  could  employ  to  assist  in  the  design  and  maintenance  of  their 
efforts. 

#7  Encourage  the  establishment  of  a  private  sector  based  "Data 
Exchange  Clearing  House"  which  compiles,  analyzes  and 
disseminates  reports  related  to  data  on  program  interventions 
and  outcomes.  This  information  will  better  enlighten  HCOP 
grantees  about  the  cognitive  and  non-cognitive  factors  that 
contribute  most  to  the  success  of  HCOP  funded  programs  in 
aiding  minority/disadvantaged  students  to  become  more 
academically  competitive  for  health  profession  school  entry, 
retention  and  graduation.  Even  tough  intuition  suggests  many  of  the 
approaches  may  be  effective,  there  is  a  need  to  confirm  these  beliefs  with  hard 
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data.  Such  a  confirmation  will  necessitate  HCOP  funded  programs  working 
cooperatively  and  systematically  together  to  compile  and  subsequently  analyze 
the  cognitive  and  non-cognitive  variables  which  influence  academic 
performance. 

#8  PDA  should  encourage  the  development  of  a  tiered  academic 
enrichment  system  that  provides  highly  motivated, 
minority/disadvantaged  students  an  opportunity  to  cultivate 
their  cognitive  skills  under  guidance  over  time.  In  addition, 
PDA  should  support  the  establishment  of  at  least  one  tiered 
program  specializing  in  cognitive  skill  development  within  each 
region  of  the  United  States.  The  focus  on  developing  and 
institutionalizing  tiered  relationships  can  ensure  that  skills  development  does 
not  end  at  the  door  of  an  individual  project. 


Chapter  II  focuses  on  learning  strategies,  particularly  those  that  incorporate 
collaborative  learning,  such  as  "Supplemental  Instruction"  (SI)  and  Video-based  assisted 
learning.  These  strategies  are  highlighted  in  this  report  because  they  have  demonstrated 
their  effectiveness  in  improving  cognitive  skills  and  in  enhancing  the  rate  of  acceptance 
and  timely  graduation  of  minority/disadvantaged  students  from  health  professions 
schools.  A  justification  is  included  which  substantiates  the  identification  of 
"Collaborative  Learning"  as  an  umbrella  approach  that  evolved  from  Piaget's6  research 
on  cognitive  thought  skills  development  and  Karplus'"  "Learning  Cycle"  which  applies 
Piaget's  conclusions  within  the  context  of  the  American  educational  system.  Included 
in  this  section  is  an  overview  of  the  categories  of  interventions  which  have  statistical 
data  supporting  their  effectiveness.  This  is  followed  by  an  explanation  of  the 
methodology  for  selecting  the  Project  Summaries  which  are  presented  in  Chapter  V. 
Lastly,  the  Findings  and  Recommendations  section  of  this  chapter  will  provide  insight 
into  the  design  of  future  HCOP  interventions. 


Jean  Piaget,  a  19th  century  Swiss  psychologist  and 
philosopher. 

Robert  Karplus ,  PhD.,  a  contemporary  professor  of 
math  and  physics  at  the  University  of  California 
at  Berkeley. 
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Findings  on  Intervention  Model 

Nearly  all  HCOP  funded  programs  provide  some  type  of  cognitive  skills  development 
support  to  participants.  Most  commonly,  cognitive  skills  development  is  encompassed 
in  the  generic  course  designations  of  a  study  skills  class  and /or  reading  class  and /or 
problem-solving  class.  An  analysis  of  48  projects  in  the  1994  Final  Report  on  the 
Evaluation  of  HCOP  Summer  Programs8  revealed  that  on  average,  only  6  hours  out  of 
the  general  40  hours  per-week  program  schedule  is  dedicated  specifically  to  the 
development  of  cognitive  skills  which  make  the  student  more  highly  competitive  in  all 
science  classes,  as  well  as,  in  standardized  test-taking  (MCAT,  SAT,  etc.).  Moreover,  the 
cognitive  skills  instructors  are  often  special  consultants  and  their  efforts  are  not  directly 
linked  to  the  science-based  course  work  which  comprises  approximately  26  hours  of  the 
weekly  program  schedule. 

In  recognition  of  the  need  to  emphasize  the  integration  of  cognitive  skills  development 
(eg.  critical  thinking,  problem  solving  and  reading  comprehension)  activities  with 
science-content  education  (eg.  course  lectures),  several  HCOP-funded  projects  have 
designed  approaches  with  this  goal  in  mind.  One  summer  program  which  has  received 
well  deserved  attention  for  its  capability  to  nurture  content  mastery  and  cognitive  skills 
development  is  Project  SOAR  (Stress  on  Analytical  Reasoning).  SOAR,  however,  is  the 
last  step  in  a  sequence  of  summer  programs  which  starts  the  summer  prior  to  entering 
the  10th  grade  of  high  school  and  ends  the  summer  prior  to  entering  the  freshman  year 
of  college.  Students  may  elect  to  participate  in  one  or  all  steps  of  the  sequence.  The 
intent  of  the  program  is  to  provide  students  with  a  "step-up"  progression  of  academic 
enrichment  experiences.  This  approach  is  consistent  with  the  position  that  cognitive 
skills  development  efforts  are  most  effective  when  in  force  over  long  spans  of  time 
within  a  structured,  supportive  environment. 

Summer  programs  and  short  term  skills  building  activities  can  be  an  ideal  setting  to 
"plant"  the  foundation  of  a  cognitive  skills  development  process  to  be  nurtured  in  an 
integrated,  organized  environment.  These  cognitive  skills  will  continue  to  improve  if 
practiced  over  time,  preferably  under-monitored  conditions.  For  example,  a  facilitator- 
led  interactive  small  group  learning  format  is  an  excellent  vehicle  for  monitoring  the 
cultivation  of  cognitive  skills.  Thus  the  findings  of  this  contract- strongly  suggest  that 
program  staff  should  be  aggressive  in  educating  students  that  their  participation  in  short 
term  program  activities  is  part  of  a  longer  term  process. 


Based  upon  literature  and  HCOP  progress  report  reviews,  HCOP  funded  programs  that 
are  highly  structured,  and  operate  throughout  the  academic  year  (eg.  postbaccalaureate 
programs),  appear  to  have  the  greatest  success  in  improving  cognitive  skills,  facilitating 
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participant  entry  and  fostering  minority/disadvantaged  student  retention  in  health 
professions  schools.  This  category  of  program  is  consistent  with  the  position  that 
content  mastery  and  cognitive  skills  development  demand  time  and  are  best  nurtured 
in  environments  in  which  the  skills  can  be  applied  within  the  traditional  curriculum 
during  the  academic  year. 

The  HCOP  funded  programs  that  best  meet  this  challenge  are  academic  year  programs 
that  emphasize  the  integration  of  cognitive  skills  building  activities  within  a  knowledge- 
based  curriculum.  A  number  of  HCOP  funded,  post-baccalaureate  programs  fall  into 
this  category.  They  have  had  a  significant  impact  on  improving  the  academic 
competitiveness  of  minority/disadvantaged  students  entering  health  professions 
students.  Current  and  past  HCOP-funded  post-baccalaureate  programs  such  as  the  13 
"Preference  Programs"9  that  received  HCOP  preference  funding  in  FY90;  and  older 
programs  such  as  Southern  Illinois  University,  Wayne  State  University,  Creighton 
University,  and  Georgetown  University  have  demonstrated  the  success  of  HCOP 
interventions10.  As  discussed  in  Chapter  5,  these  and  other  post-baccalaureate 
programs  have  been  very  successful  in  facilitating  the  entry  and  timely  graduation  of 
minority/disadvantaged  students. 


Recommendations: 


The  following  recommendations  have  an  excellent  chance  of  being  well  received  by 
HCOP  grant  applicants  because  they  place  emphasis  on  approaches  either  currently 
being  used  by  some  HCOP  grantees  or,  on  methodologies  originally  developed  with 
HCOP  financial  support. 

#1  Encourage  grantees  to  implement  learning  approaches  that  have  been  shown  to 
be  statistically  effective  in  improving  student  comprehension  and  conceptual 
understanding  of  lectures  and  textbook  material. 


#2  Encourage  all  grantees  to  become  familiar  with  collaborative  learning  approaches; 
and,  to  adopt  those  approaches  which  can  best  be  supported  by  their  existing 
resources  and  are  most   appropriate  for  the  students  targeted  by  their  program. 


9 


10 


The  Health  Careers  Opportunity  Program  announced  a  funding 
preference  for  13  targeted  post-baccalaureate  program 
grants  for  the  FY90  grant  cycle:  9  for  allopathic 
medicine,  1  each  for  dentistry  and  podiatry,  and  2  for 
allied  health. 

Proceedings   of   the   First  National   PostBaccalaureate 
Programs  Conference,  Columbus,  Ohio,  March  4-5,  1991. 
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#3  DDA  should  minimally  earmark  funds  to  support  at  least  one  Grantee  in 
each  region  that  has  as  its  objective  the  improvement  of  the  HCOP 
participant's  scores  on  the  science  and  math  portions  of  standardized  tests; 
and,  makes  a  commitment  to  use  collaborative  learning  approaches  to 
improve  basic  arithmetic  operations. 

#4  Grantees  should  be  encouraged  to  sponsor  reading  interventions  that  emphasize 
evaluation,  application,  and  the  incorporation  of  new  information. 

#5  Grantees  should  be  encouraged  to  sponsor  writing  interventions  that  emphasize 
addressing  the  challenges  that  have  been  incorporated  in  the  writing  section  of 
the  MCAT  (i.e.  to  define  the  given  statement  as  thoroughly  as  possible,  to  explore 
the  meaning  of  the  given  statement  by  giving  a  contradictory  example  and  to 
discuss  ways  to  resolve  the  conflict  between  the  initial  statement  and  the 
contradictory  example). 

#6  Grantees  should  be  encouraged  to  implement  activities  in  HCOP  programs  to 
cultivate  non-cognitive  skills,  such  as  realistic  self-appraisal  and  positive  self 
concept. 

#7  Establish  a  process  by  which  HCOP  program  participants  will  leave  the  program 
with  diagnostic  results,  specific  information  about  the  programs  learning 
approaches,  outcome  performance  and  recommendations  designed  to  enhance 
future  development  of  cognitive  skills. 


Chapter  III  looks  at  the  issues  surrounding  the  Selection  Criteria  for  use  in  identifying 
participants  for  HCOP-funded  projects.  This  section  begins  with  the  Identification  of 
Terms  and  Cognitive/Non-Cognitive  Indicators  utilized  in  the  targeting  of  students 
likely  to  benefit  most  from  a  HCOP  education  intervention.  The  chapter  continues  by 
drawing  upon  findings  that  surfaced  from  the  literature  search,  the  HCOP  Work  Group 
(HWG)  and  the  review  of  the  HCOP"  Progress  Reports11  to  show  correlations  between 
Cognitive  /Non-cognitive  Indicators  and  performance  in  science  courses  and  on 
standardized  exams  (eg.  Scholastic  Assessment  Test,  Medical  College  Admission  Test 
and  United  States  Medical  Licensure  Examination).  Special  attention  is  focused  on 
indicators  that  may  reduce,  what  the  HWG  considers,  the  excessive  emphasis  that  health 
professions  schools  place  on  minority /disadvantaged  student  performance  on 
standardized  tests.  Finally,  Recommendations  are  provided  on  which  criteria  appear  to 
be  the  most  effective  in  selecting  students. 


11  The  Division  of  Disadvantaged  Assistance  selected 
44  HCOP  Progress  Reports  that  reflected  the 
activities  of  10  individual  projects. 
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Findings  on  Participant  Selection  Process 

The  literature  is  filled  with  studies  showing  that  undergraduate  cumulative  grade  point 
averages  (UCGPA)  and  undergraduate  cumulative  science  GPAs  (UCSGPA),  together 
with  physical  and  biological  science  MCAT  sub-test  scores,  have  statistically  significant 
correlations  to  successful  performance  in  medical  school  basic  science  courses.12  13  Few 
of  these  studies  had  a  particular  focus  on  minority  students.  However,  in  a  1986  study 
conducted  at  Howard  University,  an  historically  black  university,  similar  results  were 
also  reported  for  its  largely  minority  student  population14.  One  possible  explanation  for 
the  similarity  of  results  may  rest  with  the  fact  that  Howard's  medical  school  student  body 
(as  is  true  for  all  medical  schools)  has  a  significantly  homogenous  cognitive  profile. 

It  is  notable  that  the  majority  of  educational  research  focusing  on  minority  health 
professions  students  has  been  conducted  at  predominantly  white  schools.  Sedlacek  and 
Prieto  assert  that  cognitive  variables  alone  (grade  point  average,  standardized  tests, 
course  grades)  are  not  the  best  predictors  of  academic  success  for  minority  students.15 
Their  research  indicates  that  non-cognitive  attributes  such  as  realistic  self-appraisal  and 
positive  self-concept  taken  with  selective  cognitive  variables  (such  as  GPA  and  SAT) 
provide  additional  insights  into  the  learning  strategies  of  minority  students,  and  in  turn, 
their  potential  success  in  various  academic  curricula.  Sedlacek  and  Sternberg  suggest  that 
minority  students  develop  and  foster  learning  behaviors  to  deal  with  different  societal 
challenges,  such  as  cultural  barriers,  which  majority  students  have  not  commonly 
experienced.  It  is  these  additional  adaptive  learning  strategies  which  can  contribute  to 
a  minority  student's  ability  to  succeed  in  the  university  environment.  Moreover,  in 
deference  to  the  importance  of  the  non-cognitive  variables  in  predicting  the  performance 
of  minority  students,  the  AAMC  admissions  committee  has  sponsored  several  workshops 
emphasizing  this  finding  over  the  past  two  decades.16 

The  HCOP  progress  reports  as  well  as  the  comments  expressed  by  the  HCOP  work  group 


12  Association    of    American    Medical    Colleges,      (1994).    MCAT 

Validity   Study 
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15 


16 


Brooks,  Michael  and  Jackson,  James.  (1981). 
"Validity  of  the  New  MCAT  for  Predicting  GPA  and 
NBME  Part  I  Examination  Performance" .  Journal  of 
Medical  Education.  Vol  56,  pp  767-769. 

Anderson,  J.  and  Mitchell,  K.  (1986).  Pre-admission  Data 
and  Medical  Performance  For  Black  Medical  Students,  from 
paper  presented  at  AAMC  conference. 

Sedlacek,  W.  and  Prieto,  D.  (1990)  "Predicting  Minority 
Student  Success  in  Medical  School,"  Academic  Medicine. 
65(3)  161-66. 

Sedlacek,  W.  and  Prieto.  (1982)  "An  Evaluation  of  the 
Simulated  Minority  Admissions  Exercise, "  Journal  of 
Medical   Education.  Volume  57 


participants  support  the  philosophy  that  non-cognitive  attributes  should  play  an 
important  role  in  the  assessment  of  the  relative  academic  risk  for  individual 
minority /disadvantaged  students.  However,  even  though  many  programs  have  collected 
non-cognitive  data,  few  have  analyzed  the  non-cognitive  data  as  a  basis  for  predicting 
levels  of  academic  risk. 

Our  research  led  to  the  hypothesis  that  a  more  structured  and  controlled  application  of 
cognitive  and  non-cognitive  selection  criteria  for  minority /disadvantaged  students  will 
provide  an  insightful  and  comprehensive  profile  of  indicators  useful  in  determining  a 
student's  level  of  relative  academic  risk.  Within  the  context  of  knowledge  about  the 
curricula  of  health  professions  schools,  the  profile  should  provide  a  strong  foundation  for 
determining  which  minority /disadvantaged  students,  among  the  applicant  pool,  are  most 
likely  to  be  accepted  into  health  professions  schools  and  graduate  in  a  timely  manner, 
with  the  assistance  of  a  HCOP  funded  intervention. 

Findings  relating  to  the  role  of  cognitive  and  noncognitive  variables  in  predicting 
academic  performance  must  be  considered  when  measuring  risk  levels.  Unfortunately, 
terms  such  as  "academic  risk"  and  "marginal"  student  are  not  clearly  defined.  They  often 
have  different  meanings  from  one  HCOP  grantee  admissions  committee  and /or  health 
professions  admissions  committee,  to  another.  The  research  underscores  how  "risk"  and 
"marginal"  tend  to  be  deterrnined  as  frequently  by  internal  institutional  or  program 
factors  as  it  is  by  student  cognitive  and  non-cognitive  profiles.  The  plethora  of  the  factors 
that  come  to  bear  on  defining  "academic  risk"  suggests  it  would  be  fruitless  to  try  to 
develop  a  definition  of  "academic  risk"  that  would  be  "universally"  acceptable  to  HCOP 
programs  and /or  to  health  professions  schools.  Therefore,  this  report  discusses  how  each 
HCOP  program  individually  might  attempt  to  quantify  levels  of  student  risk  on  the  basis 
of  the  following  characteristics  and  variables:  1)  known  health  professions  school  barriers, 
2)  the  program  performance  objectives,  3)  the  financial  and  non-financial  resources 
available  to  support  the  HCOP  program  and,  4)  the  cognitive  and  non-cognitive  profiles 
of  the  student  applicants. 

Once  the  participant  variables  are  described,  the  report  then  discusses  how  cognitive  and 
non-cognitive  variables  can  be  entered  into  formulae  which  has  been  generated  through 
regression  analyses.  As  a  result,  a  project-specific  HCOP  Selection  Index17  is  created  that 
quantifies  an  applicant's  relative  risk  level  in  terms  which  are  significant  to  the  individual 
program. 


17 


A  sample  Selection  Index  is  included  in  the  full  report 
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Recommenda  tions 

#1  Encourage  HCOP  Grantees  to  utilize  both  cognitive  and  non-cognitive  variables 
as  a  basis  for  the  selection  of  HCOP  participants  and  the  matching  of  those 
participants  to  the  programs  which  are  most  likely  to  benefit  them. 

#2  Grantees  should  be  encouraged  to  collect  the  "minimum  data  set"  (discussed  in 
Chapter  IV)  together  with  additional  information  from  student  applications,  letters 
of  recommendation,  by  telephone  or  personal  interview  and  when  possible  by 
means  of  pre-admission  cognitive  and  non-cognitive  assessment  tests. 

#3  While  Grantees  should  be  encouraged  to  use  non-cognitive  variables  in 
conjunction  with  cognitive  variables,  they  should  be  strongly  cautioned  on  how 
best  to  apply  this  information  within  the  emotionally  charged  context  of  the 
student  selection  process. 


Chapter  IV  explores  the  issues,  barriers  and  limitations  of  the  data  collection  efforts, 
implemented  by  past  and  present  day  HCOP  projects.  An  underlying  objective  for  all 
HCOP  funded  programs  is  to  assist  minority/disadvantaged  students  in  demonstrating 
a  level  of  knowledge  and  cognitive  skills  that  will  make  them  competitive  for  health 
professions  school  acceptance  and  graduation.  Few  question  the  fact  that  HCOP  funded 
programs  have  been  associated  with  an  impressive  increase  in  the  number  and  percentage 
of  minority  /disadvantaged  students  gaining  admission  to  health  professions  schools.  The 
challenge  facing  this  review  was  in  the  assessment  of  HCOP  Progress  Report  data  and 
the  directive  to  develop  a  comparative  analysis  which  cut  across  programs.  However, 
the  data  reported  by  each  Grantee  was  largely  presented  in  the  aggregate.  This  factor 
made  it  difficult  to  identify  the  analogous  variables  necessary  to  generate  an  effective 
review. 

There  is  great  enthusiasm  among  HCOP  Grantees  about  the  prospect  of  developing 
guidelines  and  processes  by  which  reliable,  relevant  student  level  and  program  data  can 
be  better  compiled,  analyzed  and  disseminated.  It  is  apparent  to  all  involved  in  this 
process  that  the  generation  of  reliable  analyses  assessing  the  effectiveness  of  HCOP 
funded  interventions  will  require  the  creation  of  large  common  data  sets  and  statistical 
expertise. 


Findings  from  Data  Review 

Review  of  the  literature  review  and  HCOP  Progress  Reports  revealed  that  HCOP  grantees 
collect  data  that  varies  greatly  in  terms  of  the  amount,  type  and  use.  In  addition,  HCOP 
grantees  utilize  a  wide  variety  of  statistical  reporting  formats  to  describe  participants  and 
to  support  the  claims  pertaining  to  their  effectiveness  and  general  outcomes.    The  data 
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reviewed  for  this  research  was  most  commonly  presented  in  one  or  more  of  the  following 
levels  of  analysis: 

•  Aggregate  data 

•  Cohort  data 

•  Student  specific  data 

In  understanding  the  applicability  of  project  data  to  future  programming  efforts,  these 
reporting  levels  might  be  viewed  as  part  of  a  continuum  of  analysis  useful  in  the  ultimate 
identification  of  effective  recommendations.  The  foundation  of  this  process  is  in  the 
collection  of  comparable  indicators  which  can  be  used  to  statistically  validate 
relationships  between  interventions  and  outcomes. 

Aggregate  data  is  most  often  used  to  provide  comparable  assessments  of  programmatic 
impacts.  It  offers  an  easily  interpretable,  snap-shot  of  important  program  elements  and 
is  useful  in  communicating  information  on  a  broad  spectrum  of  HCOP-supported 
strategies.  However,  this  level  of  data  does  not  demonstrate  a  clear  cause-and-effect 
relationship  between  student  profiles  and  program  interventions. 

Cohort  data  presents  information  in  groupings  by  like  variables  which  are  useful  in 
drawing  correlations  between  program  participant  skill  levels  and  the  application 
requirements  of  various  health  professions  schools.  Additionally,  cohort  data, 
consistently  captured  in  like  data  elements  lends  itself  to  constructive  comparative  and 
impact  analysis  over  the  life  span  of  individual  HCOP-funded  efforts.  This  level  of 
analysis  is  critical  in  determining  the  transferability  of  a  given  intervention  strategy. 


Student-specific  data  provides  the  greatest  insights  about  the  relationships  between 
cognitive  and  non-cognitive  variables,  the  interventions  implemented  to  impact  them 
and  the  resultant  improvements  in  the  skills  these  variables  represent.  This  data  is 
critical  in  the  initial  stages  of  intervention  design  in  that  an  accurate  needs  assessment 
of  the  target  student  population  will  lead  to  an  effective  match  of  program  activities 
and  student  needs.  Caution,  however,  must  be  taken  to  ensure  that  student-specific 
data  is  kept  confidential.  This  may  be  accomplished  through  ±he  deletion  of  names 
and  social  security  numbers  from  files  and  replacing  this  information  with,  for 
example,  HCOP  identification  codes. 

The  HCOP  Progress  Report  review  indicated  that  HCOP  projects  most  frequently 
reported  data  in  the  aggregate.  However,  the  research  findings  support  the  desirability 
of  collecting  student  level  data.  A  more  detailed  presentation  of  these  research  findings, 
statistically  validating  the  correlation  between  student  specific  data  and  intervention 
outcomes,  will  be  described  in  Chapter  4  of  this  report. 

Many  HCOP  grantees  are  limited  in  the  type,  quality  and  amount  of  student  level  data 
that  they  can  collect.    It  is  unlikely  that  these  programs  have  direct  access  to  course 
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grades  after  the  student  completes  the  program  and  continues  along  the  educational 
pathway.  Thus,  these  programs  cannot  readily  measure  the  impact  of  their  interventions 
on  actual  academic  year  course  performance.  HCOP  funded  summer  programs,  in 
particular,  will  continue  to  have  problems  tracking  the  impact  their  interventions  have 
on  subsequent  standardized  test  performance  and  health  professions  school  acceptance, 
unless  some  tracking  process  is  established  that  is  cost-effective,  formally  structured  and 
linked  to  other  learning  experiences. 

The  following  recommendations  are  based  upon  the  research  findings  and  analysis: 

Recommendations: 

#1  Develop  a  common  set  of  'confidentiality'  guidelines  and  encourage  all 
Grantees  to  collect,  analyze  and  disseminate  student  related  data  within  the 
boundaries  of  these  guidelines. 

#2  Encourage  Grantees  to  collect  cognitive  and  non-cognitive  data  utilizing  a 
limited  number  of  assessment  instruments  that  have  been  field  tested  and 
shown  to  produce  outcome  data  that  is  valid  for  minority/disadvantaged 
students. 

#3  Encourage  Grantees  to  reflect  the  components  of  a  "minimum  data  set"  in  the 
questions  they  include  on  their  participant  applications. 

#4  Establish  a  HCOP  Fellows  Program  that  identifies  the  type  of  HCOP 
program;  and  also  indicates  how  many  different  HCOP  programs  the  a 
student  has  participated  in.  The  encourage  Grantees  to  use  this  information 
to  enhance  their  ability  to  track  and  subsequently  report  on  the  academic 
progress  of  former  participants  as  they  progress  through  the  health 
profession  schools. 

#5  Encourage  Grantees  to  participate  in  a  national  program  designed  to  better 
track  the  academic  performance  and  the  ultimate  career  path  of  former 
HCOP  student  participants. 

#6  Grantees  should  be  encouraged  to  track  the  academic  performance  of  past 
participants  and  relate  academic  outcomes  to  the  non-cognitive  attributes 
defined  while  a  participant  in  the  program. 


Chapter  V  provides  a  sampling  of  academic  enrichment  programs  which  have  collected 
statistically  viable  data  linking  their  intervention  to  the  improved  critical  thinking, 
problems-solving,  and  standardized  test-taking  skills  of  the  participants.  For  the  purpose 
of  this  review,  the  programs  highlighted  in  this  section  have  been  divided  into  two 
sections,  Program  Overview  and  Program  Outcomes. 
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Approaches  and  Programs  Highlighted: 

Stress  On  Analytical  Reasoning  (SOAR) 

Supplemental  Instruction  (SI) 

FirstPrep 

Georgetown  Experimental  Medical  Studies  (GEMS) 

Video-based  Supplemental  Instruction  (VSI) 

The  Mathematics  Workshop  Program  (MWP) 


Chapter  VI  is  included  to  provide  an  easily  reference  to  the  recommendations 
incorporated  throughout  this  review.  Additionally,  this  section  provides  implementation 
guidelines  for  each  of  the  recommendations  and  includes  rationales  to  support  suggested 
enhancements  to  HCOP  operations. 

Encompassed  in  the  body  of  the  full  report  are  comprehensive  discussions  and  analyses  of  HCOP 
Interventions,  Participant  Selection  Criteria  and  Project  Data,  as  summarized  in  this  Executive 
Summary.  Preparation  of  this  report  reflects  the  in-depth  examination  of  HCOP  Grantee  progress 
reports,  an  extensive  search  of  the  educational  literature,  analysis  of  HCOP  Progress  Report  data, 
and  the  deliberations  of  the  HCOP  Work  Group. 
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CHAPTER  I 
INTRODUCTION 


The  challenging  combination  of  inferior  school  systems,  economic  hardship  and  the 
desolation  of  family  support  networks  has  led  to  the  reality  of  a  significant  minority 
population  which  is  both  economically  and  educationally,  disadvantaged.  In  conjunction 
with  this  fact,  according  to  the  AAMC1,  while  African  Americans,  Hispanics,  American 
Indians  and  mainland  Puerto  Ricans  constitute  nearly  20  percent  of  the  total  United 
States  population,  until  the  1990's  they  had  consistently  represented  less  than  ten  percent 
of  the  medical  school  matriculants. 

Begun  in  1972  as  the  Special  Health  Career  Opportunity  Grant  (SHCOG)  Program,  the 
Federal  educational  initiative  known  today  as  the  Health  Careers  Opportunity  Program 
(HCOP)  was  authorized  by  Title  VTJ,  Section  787  of  the  Public  Health  Service  Act,  as 
amended  by  Public  Law  100-607.  Since  the  initiation  of  Title  VTJ  funding  for 
minority /disadvantaged  programs,  the  number  of  underrepresented  minorities  enrolled 
in  health  professions  schools  (HPS)  has  increased  nearly  300  percent.  Over  the  past 
twenty  years  of  its  existence,  the  Health  Careers  Opportunity  Program  (HCOP)  has 
facilitated  the  entrance  and  graduation  of  thousands  of  minority/  disadvantaged  students 
into  health  professions  schools. 

From  the  literature,  the  data  confirmed  that  minority  students  do  have  an  interest  in  the 
sciences  and  in  becoming  health  professionals  (see  Volume  II  of  Final  Report,  Literature 
Search  on  Minority/Disadvantaged  Support  Programs).  The  fact  that  they  remain 
underrepresented  in  these  fields  suggests  that  even  today,  minority  students  continue 
to  encounter  factors  along  the  educational  pathway  that  dissuade  them  from  pursuing 
science  and  health  careers  or  contribute  to  their  drop  out  rate  from  the  health  science's 
pipeline. 

The  goal  of  this  research  effort  is  to  offer  HCOP  managers  additional  means  to  counter 
some  of  the  more  significant  factors  that  dissuade  minority  students  from  pursuing 
careers  in  the  health  professions.  This  effort  offers  HCOP  managers  some  strategies  and 
tools  that  can  be  useful  in  designing  future  interventions  based  on  proven  approaches 
that  will  positively  effect  the  number  of  minority /disadvantaged  students  each  year, 
who  are  accepted  into  health  professions  school  and  graduate  in  a  timely  manner. 
Additionally,  this  research  has  led  to  the  identification  of  tracking  and  evaluative 
processes  to  maximize  the  impact  of  limited  HCOP  resources.   The  analyses  presented 
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in  this  report  reflect  information  gained  through  an  expansive  literature  search,  thorough 
examination  of  progress  reports  provided  by  the  Division  of  Disadvantaged  Assistance 
(DDA)  and  the  recommendations  of  the  HCOP  Work  Group  (HWG)  which  was 
established  in  support  of  this  research  effort.  The  information  contained  in  this  report 
responds  to  Tasks  #12  and  #13  of  the  research  contract  which  mandates  the  compilation 
of  all  findings  derived  from  this  analysis  into  a  Final  Report. 

The  questions  addressed  in  this  research  contract  are: 

•  If  minority/disadvantaged  students  have  the  interest  and  motivation  to  pursue 
an  education  in  the  sciences,  and, 

•  if  financial  constraints  can  be  overcome  (or  set  aside), 

•  then,  what  interventions  are  needed  to  ensure  that  a  larger  percentage  of  these 
students  are  successful  in  entering  and  graduating  from  health  professions 
schools? 

•  What  selection  criteria  are  needed  to  ensure  that  the  students  most  likely  to 
benefit  from  these  interventions  are  identified? 

•  How  can  these  interventions  be  tracked  and  assessed  over  time  so  that  those 
components  which  are  the  most  effective  can  be  replicated  within  other  projects? 


NEW  CHALLENGES  FACING  HEALTH  PROFESSIONS  SCHOOL  APPLICANTS 

As  a  result  of  shifting  political  and  economic  priorities,  new  issues  have  arisen  for 
HCOP-funded  programs.  Factors  such  as  health  care  reform  are  causing  a  ripple  effect 
throughout  the  health  care  system.  Hospitals  and  health  agencies  are  dramatically 
reducing  their  operations,  and  this  'down  sizing'  trend  affects  the  number  of  slots 
available  for  residencies  and  entry  level  health  professional  positions.  Thus,  many  health 
professions  school  applicants  are  facing  increasing  competition  for  a  decreasing  number 
of  slots.  Within  this  environment,  HCOP-funded  programs  can  continue  to  play  a 
critical  role  in  improving  the  competitive  level  of  minority/disadvantaged  HPS 
applicants. 


Reformatting  National  Standardized  Tests 

As  the  ratio  of  the  number  of  applicants /to  the  number  of  health  professions  school  slots 
increases,  the  national  standardized  testing  formats  are  also  changing.  In  this  regard, 
standardized  tests  such  as  the  Scholastic  Aptitude  Test,  Medical  College  Admissions 
Test,  and  United  States  Medical  Licensure  Examination  are  placing  a  greater  emphasis 


on  the  test-taker's  ability  to  digest  and  analyze  detailed  reading  passages,  and  to  respond 
to  questions  through  a  process  of  deduction  and  critical  thinking. 

Due  to  many  of  the  societal  and  educational  factors  elaborated  in  Volume  II  (Literature 
Search  on  Minority  /Disadvantaged  Support  Programs)  of  this  report,  national  data 
indicate  that  a  significant  portion  of  educationally  and  economically  disadvantaged 
students,  many  of  whom  happen  to  be  minorities,  continue  to  test  lower  on  average  than 
do  most  majority  students.  For  example,  Whitten's2  research  shows  that  mean  MCAT 
scores  of  African  American  medical  school  applicants  remained  at  approximately  the 
same  for  the  period  1981-1990,  and  were  less  competitive  than  those  of  the  majority 
students. 

The  review  of  the  literature  prepared  for  this  report  leads  to  the  following  insights: 

•  First,  priorities  are  increasingly  focused  on  building  a  'pipeline'  of  young 
minority /disadvantaged  students  who  have  developed  the  cognitive  skills 
necessary  to  be  competitive  in  an  increasingly  more  challenging  market  for  health 
professions  school  slots. 

•  Second,  while  the  number  of  under-represented  minority  medical  applicants  and 
matriculants  has  increased,  the  academic  profiles  of  these  students,  as  indicated 
by  their  average  GPA  and  standardized  test  scores,  have  not  significantly 
improved.  This  may  indicate  that  as  competition  increases  for  a  more  limited 
number  of  HPS  slots  (particularly  in  the  fields  of  medicine  and  osteopathy), 
minority /disadvantaged  students  may  not  continue  to  show  the  level  of  success 
in  HPS  admissions  that  they  have  experienced  over  the  last  four  years  (see 
Appendix  2). 

•  Third,  given  these  trends,  and  until  the  'pipeline'  begins  to  yield  greater  numbers 
of  minority /disadvantaged  students  whose  cognitive  skills  (critical  thinking, 
problem  solving)  are  more  competitive,  programs  such  as  HCOP  continue  to  play 
a  critical  role  in  providing  resources  in  support  of,  a)  the  selection  and  matching 
of  students  to  programs  most  likely  to  meet  their  skills  development  needs;  b) 
quality  programs  which  provide  effective  cognitive,  curriculum-linked  skills 
development  for  students  (high  school  senior  through  HPS  matriculants);  and,  c) 
academic  year  programs,  such  as  the  post-baccalaureate  programs,  which  provide 
an  environment  for  long  term  nurturing  of  skills. 

THE  HEALTH  PROFESSIONS  PIPELINE  AND  HCOP'S  CONTINUING  ROLE 

Volume  II  of  this  report  provides  an  overview  of  a  several  strategies  designed  to  expand 


Whit ten,  Charles  F.   (1991)  .  Proceedings  of  the  First 
National  Postbaccalaureate  Programs  Conference.   86-96. 


the  number  of  minority /disadvantaged  students  in  the  health  professions  pipeline.  For 
example:  the  Association  of  American  Medical  Colleges'—  Project  "3000  by  2000,"3  calls 
for  the  development  of  several  national  networks  and  databases  to  track  programs  (as 
well  as  individual  students)  that  can  help  identify  a  critical  pool  of 
minority/disadvantaged  candidates  for  health  professions  training.  To  complement  this 
and  other  efforts,  the  National  Science  Foundation's  Urban  Systemic  Initiative  is  focused 
on  producing  dramatic  reform  in  the  delivery  of  math  and  science  instruction  at  the 
secondary  school  level  which  should  supply  the  health  professions  "pipeline"  with 
increasing  numbers  of  well-prepared  minority/disadvantaged  high  school  graduates. 

These  approaches,  however,  do  not  necessarily  respond  to  the  immediate  and 
intermediate  need  for  increased  numbers  and  more  competitive  minority/disadvantaged 
students  at  the  end  of  the  health  professions  pipeline.  The  research  findings  support  the 
continuation  of  current  programs  such  as  the  Health  Careers  Opportunity  Program 
(HCOP).  The  information  presented  within  this  report  indicates  that  many  of  these 
programs  can  provide  the  critical  element  of  assistance  needed  to  facilitate  the  increased 
success  of  minority/disadvantaged  students  in  entering  and  graduating  in  a  timely 
manner  from  health  professions  schools. 

The  subsequent  chapters  of  this  report  present  the  findings  and  recommendations 
derived  from  the  research  which  may  provide  a  foundation  for  the  enhancement  of 
HCOP.  Cited  at  the  beginning  of  each  chapter  are  those  Tasks,  dictated  by  the  Research 
Contract,  to  which  the  analysis  responds.  Included  in  these  sections  is  a  description  of 
each  Task,  as  it  appears  in  the  contract  Scope  of  Work.  The  following  paragraphs 
summarize  the  key  elements  of  each  aspect  of  the  analysis  introduced  in  this  report: 

•  Chapter  II,  Effective  Learning  Approaches:  'Improving  Cognitive  Skills'  in 
Minority/Disadvantaged  Student  Populations 

This  Chapter  presents  results  of  the  analysis  of  various  educational 
approaches  studied  in  preparation  for  this  report.  The  research,  together 
with  recommendations  provided  by  the  HCOP  Work  Group  (HWG)  and 
information  gleaned  from  selected  HCOP  progress  reports,  presents 
insights  on  a  spectrum  of  collaborative  learning  approaches.  The 
approaches  highlighted  in  this  section  of  the  report  have  been  associated 
with  improvements  in  cognitive  skills  as  measured  by  increases  in 
academic  year  math  and  science  course  performance.  Chapter  II  reflects 
the  product  of  Tasks  #10  and  #11  through  the  presentation  of  intervention 
approaches  which  have  been  identified  as  the  result  of  the  application  of 
the  evaluative  methodologies. 

•  Chapter  III,  Selection  of  the  HCOP  Participant 
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This  Chapter  discusses  the  issues  involved  in  the  identification  of  students 
most  likely  to  benefit  from  HCOP  and  the  need  for  effectively  matching 
student  needs  with  the  programs  most  likely  to  meet  those  needs.  The 
recommendations  provide  a  foundation  upon  which  individual  programs 
can  develop  participant  selection  indices  which  best  suit  their  needs.  In 
response  to  Tasks  #6  and  #8  of  the  Scope  of  Work,  Chapter  HI  presents 
proposed  HCOP  participant  selection  criteria,  and  approaches  for  program- 
specific  application  and  testing  of  these  criteria. 

Chapter  IV,  Assessment  of  HCOP  Project  Data 

Analysis  of  available  HCOP  project  data,  together  with  a  review  of  the 
educational  literature,  points  to  a  need  for  consistency,  between  HCOP 
projects,  in  the  quality  and  depth  of  data  collected.  Reliable  data  provides 
the  fuel  for  the  selection  of  participants,  the  evaluation  of  interventions  and 
the  translation  of  quality  models.  This  Chapter  speaks  to  the  issues 
surrounding  the  development  and  application  of  analytical  methodologies 
in  assessing  the  outcomes  of  previous  and  existing  HCOP  projects.  The 
information  illustrated  in  this  section  relates  to  the  products  requested 
under  Tasks  #5,  #10  and  #11. 

Chapter    V,    A    Sampling    of    Interventions     Designed    to    Improve    the 
Competitiveness  of  Minority/Disadvantaged  Students 

This  Chapter  presents  a  selection  of  programs  and  approaches  which  have 
shown  statistically  validated  positive  correlations  between  their 
intervention  and  minority  /disadvantaged  student  college  and  Health 
Professions  School  achievement.  This  Chapter  is  a  product  of  the  literature 
review  directive  outlined  under  Task  #2. 

Chapter  VI,  Comprehensive  Listing  of  Recommendation,  Related  Guidelines 
and  Rationales 

The  last  Chapter  provides  a  comprehensive  listing  of  all  the 
recommendations,  incorporated  through  out  this  report,  into  one  concise 
reference  guide.  Included  as  well,  are  what  the  HCOP  Work  Group 
(HWG)  believed  to  be  the  key  guidelines  which  will  serve  as  a  'road  map' 
for  the  enhancement  of  HCOP  efforts.  Lastly,  each  recommendation  is 
followed  by  a  rationale  detailing  the  history  and  goals  which  have 
supported  the  development  of  each  suggested  addition  to  the  program. 
Chapter  VI  is  a  combination  of  the  specific  directive  of  Task  #9  (the 
development  of  student  selection  guidelines)  and  the  product  of  the  full 
research  effort. 


CHAPTER  II 

Effective  Learning  Approaches:  'Improving  Cognitive  Skills'  in 
Minority/Disadvantaged  Student  Populations 


Chapter  II  reviews  the  learning  interventions  utilized  by  HCOP  Grantees  to  improve  the 
academic  performance  and  retention  of  minority /disadvantaged  students  in  health 
professions  schools.  This  review  is  in  concert  with  the  goals  of  the  Division  of 
Disadvantaged  Assistance  (DDA)  in  identifying  approaches  most  likely  to  effectively 
employ  limited  resources  in  assisting  minority /disadvantaged  students  to  become 
competitive  for  health  professions  school  acceptance  and  timely  graduation.  As  outlined 
in  the  introduction,  this  portion  of  the  report  addresses  the  deliverables  as  directed 
under  Tasks  #10  and  #11  of  the  "Research  on  Minority/Disadvantaged  Support 
Programs"  project.  These  Tasks  involve  the  development  of  methodologies  for  the 
assessment  of  HCOP  funded  intervention  approaches  and  the  subsequent  analysis  of 
these  projects. 

This  review  began  under  the  directive  to  "analyze  the  effectiveness  of  specific 
intervention  approaches  in  terms  of  their  ability  to  improve  problem-solving,  reading, 
conceptualization,  and  test-taking  skills."  However,  insights  from  the  educational 
literature,  grantee  progress  reports  and  the  HCOP  Work  Group  (HWG)  revealed  that  the 
present  compilation  of  HCOP  project  data  does  not  provide  a  sufficient  basis  for  the 
breath  of  analysis  necessary  to  fully  comply  with  the  objectives  delineated  under  Tasks 
#10  and  #11.       Three  major  factors  inhibited  this  assessment: 

•  Measuring  true  gains  in  cognitive  skills,  such  as  problem-solving,  is  an  extremely 

complicated  task  involving  the  interplay  of  a  multitude  of  thinking  processes. 
The  intricacy  of  this  interaction  is  effectively  illustrated  in  Champagne's1 
"Taxonomy  of  Cognitive  Skills."  (see  Appendix  3).  Because  of  their  complex 
nature,  substantial  gains  in  these  skills  areas  are  more  likely  to  develop  over  time. 
As  a  result,  it  is  difficult  to  assess  the  impact  of  short  term  interventions  on  long 
term  cognitive  skills  development. 
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•  HCOP  project  data  were  not  configured  in  a  manner  to  facilitate  a  statistically 
valid  analysis  of  the  impact  of  the  approaches,  both  within  a  specific  program  and 
between  programs.  For  example,  because  no  minimum  data  set  exists  (such  as 
the  concept  of  a  minimum  data  set  presented  in  Chapter  IV),  there  was  little 
uniformity  in  the  outcome  measures  utilized  by  programs  to  determine  the 
effectiveness  of  an  approach.  This  range  in  outcome  measures  impeded  a 
statistically  valid  comparison  across  programs. 

•  HCOP  Project  Progress  Reports  and  impact  assessments  infrequently  presented 
data  that  was  at  the  student-specific  or  even  cohort  level  which  would  have 
allowed  for  accurate,  individual  analysis.  Without  primary  data  at  this  level, 
researchers  had  great  difficulty  in  identifying  the  factors  which  led  to  the 
program's  success  and /or  individualized  student  improvement. 

All  of  these  factors  stem  from  the  simple  reality  that  few  if  any  of  the  programs  studied 
for  this  analysis  implemented  their  approaches  as  if  they  were  a  "clinical  trial".  A 
detailed  quantitative  analysis,  which  includes  a  "control  group"  comparison,  would 
involve  a  reassessment  of  the  resources  available  to  the  projects,  given  their  overall 
program  goals  and  the  complexity  of  the  skills  being  developed. 

The  remaining  sections  of  this  chapter  are  going  to  address  program  approaches  that 
were  designed  to  meet  the  needs  most  consistently  identified  with 
minority/disadvantaged  students  and  will  provide: 

•  The  Evolution  of  Effective  Learning  Strategies. 

•  Insights  on  the  "Deficit  Learning"  approach,  a  key  element  in  many 
supplementary  educational  programs. 

•  Overview  of  the  approach  frequently  highlighted  within  the  literature  as 
one  of  the  most  promising  in  the  development  of  cognitive  skills.  This 
approach,  which  evolved  from  the  early  theories  of  Piaget  and  Karplus, 
is  generally  acknowledged  as  a  basis  for  the  development  of  various 
Collaborative  Learning  Strategies. 

•  Sample  applications  of  collaborative  learning  within  Conceptual  Models. 

•  Major  Findings  and  Conclusions  based  on  assessment  of  the  educational 
literature,  HCOP  Progress  Reports  and  the  recommendations  of  the  HCOP 
Work  Group. 


EVOLUTION  OF  EFFECTIVE  LEARNING  STRATEGIES 

As  discussed  in  the  findings  section  of  the  Literature  Search  (Volume  II  of  this  report), 
research  has  shown  that  the  underdevelopment  of  cognitive  skills  can  lead  to  academic 
difficulty  in  advanced  mathematics  and  science  courses  at  all  educational  levels.2  3  4  5 
The  educational  literature  highlighted  the  fact  that  a  significantly  greater  percentage 
of  minority  /disadvantaged  students  were  educated  in  school  systems  which  lacked  the 
ability  to  effectively  develop  the  cognitive  skills  necessary  "for  absorbing  science 
concepts,  for  organizing  them  into  a  conceptual  frame  work,  and  for  applying  them  in 
appropriate  situations."6  It  is  logical  to  deduce  therefore,  that  educational  approaches 
which  target  the  improvement  of  cognitive  skills  may  show  a  corresponding  increase  in 
academic  performance.  Based  upon  this  theoretical  premise,  this  chapter  focuses  on 
those  approaches  which  are  most  likely  to  enhance  the  academic  performance  (academic- 
year  math  and  science  grades,  and  standardized  test  scores)  and  the  retention  rates  of 
minority  /disadvantaged  students. 

To  provide  an  overview  on  the  linkage  between  cognitive  skills  and  academic 
performance,  the  following  paragraphs  highlight  Jean  Piaget's  principles  for  skills 
development  and  Robert  Karplus'  use  of  these  principles  in  developing  a  learning 
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strategy  to  improve  academic  performance  of  students. 

According  to  the  Swiss  psychologist  Jean  Piaget,  the  higher  order  skills  needed  to  solve 
complex  science  problems  are  thinking  skills.  Piaget  describes  highly  developed 
thinking  skills  as  formal  thought  or  the  level  when  an  individual  is  "...able  to  reason 
abstractly  —  to  hypothesize  and  work  from  the  abstract  to  the  particular."  This  is 
consistent  with  views  of  today's  educators  who  claim  that,  "...few  minority  students  have 
the  requisite  higher  order  skills  needed  to  [successfully]  pursue  professional  degrees, 
particularly  in  the  scientific  and  technical  fields."7 

Robert  Karplus,  Ph.D  at  the  University  of  California  at  Berkeley  was  prominent  in 
developing  an  educational  strategy,  the  "Learning  Cycle,"  noted  for  putting  Piaget's 
theories  to  work  in  the  college  classroom.  The  "Learning  Cycle"  is  based  on  "Piaget's 
idea  that  knowledge,  to  be  useful  for  reasoning,  must  be  constructed  ...  actively  by  the 
learner,"8  and  that  teachers  and  peers  play  an  important  role  in  the  process.  The  three 
phases  of  the  cycle  include: 

•  Exploration  in  which  the  students  leam  through  their  own  actions  and 
reactions  by  exploring  new  materials,  ideas,  and  relationships  with 
minimal  guidance  or  expectation  of  accomplishment.  Through  questions 
and  suggestions,  the  teacher  can  help  students  relate  the  new  experience 
to  their  existing  knowledge. 

•  Concept  Introduction  in  which  a  new  concept  or  principle  requires  the 
student  to  apply  new  reasoning  patterns  to  their  experiences.  The  concept 
may  be  introduced  by  the  teacher  and  should  always  follow  exploration 
and  relate  directly  to  exploration  activities. 

•  Concept  Application  in  which  the  students  apply  the  new  concept  and/or 
reasoning  pattern  to  additional  examples.  This  phase  is  essential  to 
extending  the  range  of  applicability  of  the  concept  and  to  help  master  the 
new  reasoning  pattern.  - 

Thus,  the  "Learning  Cycle"  provides  for  active  participation  by  the  learners  during 
exploration  and  application,  supplemented  with  instruction  by  the  teacher  during  the 
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concept  introduction  phase."9  Group  activities  are  used  in  conjunction  with  the 
"Learning  Cycle"  format  to  enable  students  to  compare  ideas,  clarify  misconceptions,  and 
provide  a  supportive  venue  for  multiple  viewpoints  and  questions.  Teachers  have  an 
opportunity  to  observe  students  throughout  the  cycle  to  evaluate  the  students' 
understanding  and  resolve  pending  difficulties  during  individualized  student-teacher 
conferences. 

Since  the  mid-1970s,  a  number  of  other  educators  developed  programs  that  used  the 
"Learning  Cycle"  strategies  in  the  classroom.   Some  of  the  better  known  programs  are: 

•  ADAPT  (Accent  on  Developing  Abstract  Processes  of  Thought) 
implemented  at  the  University  of  Nebraska, 

•  DOORS  (Development  of  Operational  Reading  Skills)  developed  by  Illinois 
Central  College. 

•  STAR  (Steps  to  Abstract  Reasoning)  created  by  Metropolitan  State  College 
in  Denver. 

All  of  these  programs  have  shown  statistically  significant  outcomes  in  terms  of 
competitive  course  performance,  high  retention  rates  and  timely  graduation.  It  is 
important  to  note  that  none  of  these  programs  was  designed  specifically  to  meet  the 
needs  of  minority /disadvantaged  students. 

In  regards  to  the  theoretical  premise  established  at  the  beginning  of  this  section,  effective 
learning  strategies  which  target  improved  grades  and  higher  retention  rates  for 
minority /disadvantaged  students  may  enhance  the  development  of  cognitive  skills 
through: 

•  their  willingness  to  participate  in  regulated,  monitored  practice  of 
integrating  information  and  concepts  to  solve  complex  math  and  science 
problems; 

•  the  process  of  group  interaction  to  affect  a  real  improvement 
in  critical  thinking,  problem  solving  and  test-taking  skills 
over  time; 

•  the  provision  of  frequent  positive  feed-back  (verbal  and 
written)  to  enhance  confidence  and  self-esteem  and  the  desire 
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to  achieve  competitive  course  grades  and  standardized  test 
scores;  and, 

•  the  process  of  empowering  students  to  employ  critical  thinking 

rather  than  making  them  dependent  on  tutoring.  This  may  be 
particularly  beneficial  if  one  accepts  the  research  that  suggests 
tutoring  relationships  do  not  effectively  promote  transfer  of  needed 
academic  skills.10 

The  literature  review  revealed  that  some  approaches  designed  to  address  the  needs  of 
minority/disadvantaged  students  are  inclined  to  be  content-specific  rather  than 
providing  interventions  which  link  skills  development  to  content  areas.  In  many  cases, 
this  perspective  has  resulted  in  the  use  of  deficit-focused,  learning  approaches. 


DEFICIT  LEARNING 

As  noted  earlier,  some  developmental  skills  programs  either  explicitly  or  implicitly 
emphasize  a  deficit-focused  delivery  model.  Their  short  term  goal  is  in  teaching 
students  the  specific  skills  that  they  have  not  yet  mastered  (i.e.  note-taking,  how  to 
recognize  the  'main  idea'  in  a  reading  passage).  For  example,  the  reading  teacher's  goal 
becomes  student  improvement  on  a  standardized  reading  test  such  as  the  Nelson-Denny 
Reading  Test.  Unfortunately,  many  students  are  able  to  gain  skills  sufficient  to  pass  a 
skills  development  course  but  do  so  without  developing  the  more  intricate  skills  of 
advanced  cognitive  reasoning  which  will  be  critical  to  competitive  academic  performance 
in  health  professions  schools.11 

Additionally,  the  deficit  learning  model  programs  often  carry  with  them  the  negative 
social  perception  that  students  participating  in  them  are  "lacking'  in  their  ability  to  hold- 
their-own  with  the  general  population.  As  a  result,  students  may  develop  a  sense  of 
embarrassment  that  hinders  their  ability  to  both  benefit  from  the  program  and  go  on  to 
be  more  successful  in  the  general  academic  environment. 
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The  educational  literature  identifies  a  need  for  redirection  of  efforts  embodied  in  the 
developmental  skills  approach  into  an  approach  which  is  more  closely  aligned  with 
academic  year  curricula.  The  following  two  models  provide  an  insight  into  how 
developmental  programs  could  evolve  into  approaches  more  likely  to  enhance  the  higher 
levels  of  cognitive  thought  encompassed  within  deductive  reasoning  and  problem 
solving.  One  of  these  approaches  is  identified  as  the  "simulations  model"12 13  in  which 
texts  and  course  materials  from  academic  year  courses  provide  the  subject  matter  upon 
which  the  skills  development  course  is  taught.  This  allows  students  to  practice  new 
techniques  using  topic  areas  which  are  "real"  or  applicable  to  their  own  studies.  The 
second  approach  involves  more  structured,  supplemental  instruction14  which  ties  skills 
development  to  credit  bearing,  content-based  courses.  In  practice  this  means  that 
students  desiring  skills  development  receive  the  same  course  matter  as  traditional 
students,  but  it  is  delivered  in  a  format  more  conducive  to  the  development  of  the  skill 
which  is  lacking.  Students  attending  the  expanded  course  format  take  exams  on  the 
same  cycle  as  the  traditional  students  but  in  reflection  of  additional  time  expended, 
students  in  the  supplemental  instruction  course  often  get  additional  credit  for  the  course. 
Both  of  the  approaches  outlined  here  relate  to  collaborative  learning  as  detailed  in  the 
next  section  of  this  report. 


OVERVIEW  OF  COLLABORATIVE  LEARNING 

The  educational  literature,  HCOP  progress  reports,  and  observations  recorded  by  the 
HCOP  Work  Group  (HWG)  strongly  suggest  that  the  collaborative  learning  strategy  is 
the  most  promising  learning  approach  to  improve  the  cognitive  skills  and  academic 
performance  of  minority /disadvantaged  students.  Its  concept  is  embodied  in  the  idea 
that  students  who  study  together,  participate  in  group  discussions  and  whose  skills  are 
developed  in  conjunction  with  the  study  of  what  is  "real"  to  them  (subject  matter  which 
is  their  area  of  academic  concentration)  are  more  likely  to  feel  positive  about  themselves, 
their  studies,  and  more  likely  to  internalize  what  they  have  learned. 

This  strategy  has  its  roots  in  the  theories  of  Piaget  and  Karplus  discussed  previously. 
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Elements  of  collaborative  learning  effective  for  health  professions  success  include: 

•  a  type  of  group  study  frequently  led  by  a  facilitator; 

•  integration  of  cognitive  skills  development  activities  with 
math  and  science  content  domains; 

•  the    nurturing    of    cognitive    skills,    in    conjunction    with 
academic  year  course  work,  over  time. 

Each  of  these  elements  multiplies  the  effects  of  the  student's  learning  experience  beyond 
the  input  of  the  initial  program.  Many  studies  have  proven  that  interactive  groups 
contribute  a  social  element  to  the  process  which  enhances  the  learning  experience.15 
The  need  to  integrate  cognitive  skills  development  with  the  subject  matter  has  been 
elaborated  by  James  Greeno  and  the  Institute  for  Research  on  Learning.  He  explains  that 
"the  ability  [for  complex  thought]  grows  through  a  progression  of  activities  that  increase 
in  their  difficulty,  but  they  involve  thinking  from  the  beginning  rather  than  having  the 
opportunity  to  think  withheld  until  the  inventory  of  alleged  materials  for  thinking  has 
been  stockpiled."16  Within  the  context  of  higher  level  thought  involving  deduction  and 
independent  thinking,  this  quote  elaborates  on  the  continuous  need  to  provide 
opportunities  for  the  reinforcement  of  cognitive  skills  through  out  the  student's  process 
of  developing  scientific  and  health  professions  content  knowledge. 

In  light  of  these  considerations,  educators  recommend  that  academic  enrichment 
programs  designed  to  enhance  the  academic  performance  of  minority/ disadvantaged 
students  should  emphasize  approaches  that  integrate  thinking  and  learning  skills 
together  with  course-specific  content  -  this  is  a  basic  element  of  collaborative  learning 
that  may  be  easily  adapted  to  HCOP  summer  and  academic  year  programs.  As  will  be 
shown  in  Chapter  V,  there  are  several  HCOP-funded  programs  utilizing  elements  of 
collaborative  learning  that  have  been  clearly  linked  to  positive  student  outcomes: 
competitive  course  grades,  greater  academic  persistence  and  increased  acceptance  rates 
to  health  professions  schools.  Moreover,  these  outcomes  are  supported  by  valid  and 
reliable  statistical  data. 


15 


16 


Clark,  MJ  &  Halpern,  DF  (1990) .  The  million  dollar 
question:  Can  an  intensive  learning  experience  improve 
success  for  the  lowest  cruartile  students? 

Greeno,  JG  (1992).  "Mathematical  and  Scientific  Thinking 
in  Classrooms  and  Other  Situations"  Enhancing  thinking 
skills  in  the  sciences  and  mathematics. 

14 


CONCEPTUAL  MODELS  FOR  COLLABORATIVE  DEVELOPMENT 

Collaborative  learning  models  can,  and  do,  vary  greatly  in  terms  of  format,  size  of 
student  groups  and  in  how  the  learning  sessions  are  conducted.  Although  variations  of 
this  approach  are  used  at  all  points  along  the  educational  pipeline.  The  following  are 
examples  of  some  of  the  more  commonly  used  models. 

MODEL  #1:  Course-Linked  Supplemental  Instruction 

This  model  begins  with  the  identification  of  specifically  targeted  courses  which  serve  as 
"gatekeepers"  —  courses  with  which  significant  numbers  of  students  encounter  difficulty. 
Examples  might  include  organic  chemistry,  physics  or  physical  chemistry. 

Support  of  the  faculty  and  administration  has  proved  to  be  essential  to  the  long  term 
survival  of  the  program.  The  faculty  member  who  instructs  in  this  "historically  difficult" 
course  contributes  materials  and  substantive  support  to  a  trained  facilitator.  This 
facilitator  is  likely  to  be  a  graduate  student  or  an  honor  student  in  the  discipline. 

Student  interaction  centers  on  the  application  of  learning  strategies  to  content  domains 
highlighted  in  lectures,  seminars,  or  required  readings.  Snap  shots  of  focused,  critical 
areas  of  the  course  (as  identified  by  the  faculty  member)  are  organized  and  presented 
in  a  manner  which  assists  the  student  group  to  address  the  material  critically,  and  to 
deduce  both  the  essential  content  and  the  study  methods  which  are  best  calculated  to 
produce  an  end-of-term  successful  grade. 

The  overt  curriculum  in  course-linked  supplemental  instruction  is  the  content  being 
covered  in  the  text  and  lectures.  The  less  evident  curriculum  addresses  the  most 
appropriate  study  methods  to  be  used.  Even  less  evident  is  the  agenda  which  aims  to 
help  students  to  understand  the  ethos  of  higher  education  and  to  adapt  their  behavior 
(academic  and  social)  to  the  model  which  prevails  on  the  campus  and  in  the  discipline 
where  they  have  elected  to  pursue  their  career  goals. 

Student  participation  in  the  supplemental  learning  program  is  typically  voluntary,  and 
usually  no  preselection  process  is  employed  to  identify  those  most  likely  to  benefit  from 
this  approach.  When  this  method  is  used  to  target  a  specific  student  group,  however, 
attendance  requirements  may  be  desirable. 

The  model  seeks  to  integrate  the  academic  content  with  skill  development  and  social 
interaction  around  the  nucleus  of  the  content  of  the  course.  Group  size  can  range  from 
2-25  students.  Programs  are  usually  connected  to  the  academic  year  curriculum, 
although  some  advanced  degree  or  specialized  exam-prep  programs  may  modify  the 
approach  to  meet  the  requirements  of  the  curriculum. 
Currently,  this  model,  or  some  variation  of  it,  is  used 
by  over  500  institutions  throughout  the  United  States  and  overseas. 
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MODEL  #2:  Small  Group  Interactive  Learning 

In  this  model,  the  learning  approach  is  implemented  in  small  groups  (6-8  students) 
where  participation  is  often  voluntary  but,  on  occasion,  is  mandated  depending  upon 
individual  program  goals.  Within  this  context,  the  groups  can  be  led  by  a  trained 
facilitator.  A  diverse  student  mixture,  in  academic  capability,  gender  and  ethnicity,  is 
desirable.  Student  profiles  are  assessed  through  the  application  of  selection  processes 
like  those  elaborated  in  Chapter  HI  of  this  report.  The  assignments  of  students  by 
faculty  or  staff  to  learning  groups  encourages  an  advantageous  mixture  of  students, 
particularly  when  based  on  educational  diversity.  When  group  assignments  are  left  to 
the  student,  the  result  is  too  often  "birds  of  a  feather  grouping  together"  i.e.  the  stronger 
students  selecting  stronger  students  and  leaving  the  weaker  students,  often  intimidated 
by  possible  rejection,  to  group  together  by  default.  This  approach  is  designed  to  be 
complementary  and  conducive  to  the  interactive  learning  process.  Study  agendas  are 
set  either  by  the  staff  or  the  trained  facilitator. 

One  adaptation  of  this  model  involves  the  addition  of  a  video  tape  of  a  professor's 
lecture  which  focuses  on  the  group's  academic  concentration.  The  video  tape  often 
includes  a  presentation  of  selected  topic  areas  and  relevant  questions  for  discussion  by 
the  group.  The  benefit  of  the  video  is  that  it  allows  the  group  to  learn  at  its  own  speed, 
i.e.  -  the  group  can  stop,  start  and  replay  the  tape,  depending  on  the  its  need  for 
reinforcement. 

An  increasing  number  of  U.S.  medical  schools  are  utilizing  video-based  assisted  learning 
to  prepare  students  for  the  United  States  Medical  Licensure  Examination  (USMLE).  This 
collaborative  learning  approach  is  designed  to  fit  within  the  6  to  8  week,  post-second 
year  period  that  medical  students  are  allotted  to  prepare  for  the  USMLE.  In  1993, 
twenty  medical  schools  used  the  program,  and  25  were  scheduled  to  use  it  in  1994. 

The  small  group  interactive  approach  which  utilizes  targeted  subject,  video-taped 
lectures  is  particularly  adaptable  to  the  scheduling  flexibility  inherent  in  HCOP  summer 
programs  and  reduced  course  load  academic  year  programs.  This  approach  is  most 
effective  when  used  either  in  conjunction  with  a  currently  attended  course,  or  after  the 
completion  of  the  course  as  a  review  tool. 


MODEL  #3:  Pair  Problem-Solving 

Perhaps  the  easiest  and  most  cost-effective  collaborative  learning  model  is  based  on  a 
"pair"  group  structure.  A  number  of  HCOP  funded  programs  have  used  this  approach 
which  is  advocated  by  Arthur  Whimbey,  Ph.D.  who  refers  to  it  as  the  "pair  problem 
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solving"  model.17  This  model  calls  for  one  student  to  be  the  problem-solver  and  a 
second  student  to  be  a  listener.  The  listener's  job  is  to  ask  probing  questions  and  to 
listen  to  the  thinking  process  of  the  problem  solver.  Pair  problem  solving  not  only 
assists  students  to  better  analyze  and  solve  problems,  but  it  also  helps  students  to 
understand  each  other  and  brings  them  closer  together.  In  addition,  it  helps  students 
to  better  understand  their  own  reasoning  processes  and  those  of  their  partners.  This 
model  can  be  readily  applied  within  summer  and  academic  year  programs. 


Group  Dynamics 

It  should  be  noted  that  most  forms  of  collaborative  learning  place  emphasis  on  having 
a  facilitator  or  group  leader  present.  This  is  felt  to  be  a  necessary  part  of  the  process  in 
order  to  keep  the  group  on  task  with  the  content  material  and  to  guide  the  development 
of  appropriate  learning  skills  which  the  students  can  adopt  and  use.  Other  models  place 
emphasis  on  forming  study,  cluster  groups  and  then  releasing  them  to  perform  in  an 
unsupervised  but  collaborative  study  environment.18 


MAJOR  FINDINGS,  CONCLUSIONS  AND  RECOMMENDATIONS 

The  educational  literature,  the  HCOP  progress  reports,  and  the  deliberations  of  the 
HCOP  Work  Group  collectively  highlight  the  complex  challenge  of  assessing  the 
effectiveness  of  a  given  HCOP  approach  in  improving  the  academic  performance  and 
cognitive  skills  of  minority /disadvantaged  students.  This  challenge  is  anchored  in  the 
fact  that,  taken  as  a  group,  HCOP  programs  employ  a  broad  spectrum  of  educational 
approaches  and  learning  approaches  to  address  a  common  goal.  This  shared  educational 
goal  is  to  improve  math  and  science  content  mastery  of  HCOP  participants  to  a  level  that 
makes  them  competitive  for  health  professions  school  entry  and  timely  graduation. 

The  findings  deemed  most  significant  were  those  that  were  statistically  related  to  the 
attainment  of  competitive  academic  year  course  grades  and  competitive  standardized  test 
scores.  In  those  cases  where  validated  data  were  provided,  collaborative  learning 
approaches  played  a  key  role  in  the  program's  success. 


17  Whimbey,  Arthur  and  Lochhead,  Jack.  (19  84) . 
Beyond  Problem  Solving  and  Comprehension. 
Lawrence  Erlbaum  Associates:  Hillsdale,  NJ. 

18  Johnson,  D.W.  et  al  (1991) .  "Cooperative 
Learning:  Increasing  College  Faculty 
Instructional  Productivity."  ASHE-ERIC  Higher 
Education  Report.  No . 4 .  Washington,  DC:  The 
George  Washington  University,  School  of 
Education  and  Human  Development 
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The  1994  Evaluation  of  HCOP  Summer  Programs19  indicated  that  relatively  few  projects 
emphasized  the  use  of  the  key  elements  of  collaborative  learning.  For  example,  only  27 
percent  —  or  less  than  1/3  —  of  the  summer  projects  used  a  small  group  interactive 
learning  approach  in  any  aspect  of  their  intervention  programs;  and,  only  11  percent 
used  small  interactive  groups  as  the  primary  learning  approach. 

Of  the  total  number  of  HCOP  Progress  Reports  reviewed  for  this  research,  only  two 
listed  collaborative  learning  strategies  as  a  program  priority: 

•  The  first  approach  was  integrated  as  part  of  a  summer 
program  that  provided  statistically  significant  data  linking 
the  achievement  of  higher  than  expected  grades  in  math  and 
science  courses  with  participation  in  the  program.  For 
example,  the  outcome  data  of  one  of  the  HCOP  Progress 
Reports  supports  the  positive  correlation  between 
participation  in  the  enrichment  experience  and  a  significant 
reduction  in  the  attrition  of  minority  students  in  two 
freshman  courses  —  pharmacy  calculation  and  medicinal 
chemistry.  Notably,  this  HCOP  program  used  a  collaborative 
learning  approach  based  on  Whimbey  and  Lockhead's 
textbook,  "Problem  Solving  and  Comprehension."  The 
Whimbey  approach  has  been  field  tested  and  is  utilized  by 
educators  as  an  effective  means  of  increasing  students'  ability 
to  analyze  problems,  and  better  comprehend  what  they  read 
and  hear.  The  project  integrated  math  and  science  content 
with  problem  solving  exercises,  and  many  of  the  students 
participated  in  these  exercises  over  four  consecutive 
summers. 

•  The  second  approach  was  implemented  within  the 
curriculum  of  a  HCOP-funded  post  baccalaureate  program. 
This  project  linked  course  content  with  a  supplemental 
instruction  session.  The  approach  was  designed  to 
supplement  difficult  medical  school  courses.  The  data 
produced  from  this  effort  indicated  that  collaborative 
learning  played  a  statistically  significant  role  in  assisting 
medical  students  who  participated  in  HCOP-funded 
postbaccalaureate  programs  to  achieve  passing  grades  in 
several  difficult  basic  science  medical  school  courses. 
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The  HCOP  Progress  Report  review  and  HCOP  Work  Group  (HWG)  discussions  surfaced 
several  summer  programs  that  incorporated  the  Whimbey  Problem-Solving  and 
Comprehension  methodology  in  their  respective  curricula.  These  efforts  however,  unlike 
those  of  the  project  (highlighted  above)  that  involved  its  students  in  collaborative 
exercises  over  several  successive  summers,  were  not  directly  linked  to  documented 
improvements  in  academic  year  grades  or  standardized  test  scores.  These  programs  did 
report  positive  outcomes  in  terms  of  post-intervention  test  scores  that  were  significantly 
higher  than  pre-intervention  test  scores.  In  addition,  they  reported  that  participants  also 
demonstrated  increased  confidence  in  their  problem-solving  abilities. 

The  researchers  additionally  noted  that  several  HCOP  programs,  that  did  not  utilize  the 
Whimbey  or  Supplemental  Instruction  approaches,  also  claimed  positive  outcome  based 
on  post-intervention  test  scores  that  were  significantly  higher  than  pre-intervention  test 
scores.  In  contrast  to  the  programs  that  linked  skill  development  to  content,  however, 
this  group  of  programs  did  not  provide  statistically  significant  data  correlating  their 
learning  approach(s)  to  improved  post  program  academic  year  grades  or  competitive 
standardized  test  scores. 

The  reinforcement  of  cognitive  skills  development  over  time  is  embodied  in  the  learning 
strategies  of  Project  3000  by  2000  and  HeadStart.  This  same  philosophy  was  espoused 
by  members  of  the  HCOP  Work  Group  (HWG).  In  concert  with  this  view,  individual 
HCOP  projects  should  collaborate  beyond  the  schools  in  which  they  are  located.  Ideally, 
it  would  be  desirable  to  build  regionalized,  "vertical  HCOP  systems."  These  systems 
should  support  the  continued  reinforcement  of  cognitive  skills  as  participants  progress 
through  post-secondary,  undergraduate  and  graduate  levels  of  education. 

Conclusion 

Collaborative  learning  is  an  important  strategy  because  it  provides  or  enhances  key 
academic  skills  while  empowering  students  to  self-initiate  critical  thinking  and  to  share 
their  knowledge  and  insights  with  others  in  order  to  promote  further  learning.  The 
small  group  interactive  format  associated  with  many  collaborative  learning  approaches 
has  been  shown  to  be  an  effective  motivating  tool  which  is  well  received  by  students. 


Students  "who  form  study  groups  report  that  they  both  enjoy  their 
work  more,  and  feel  they  learn  more,  because  of  the  academic 
discussions  within  these  groups.... In  every  comparison  of  how  much 
students  learn  when  they  work  in  small  groups  versus  when  they 
work  alone,  small  groups  show  the  better  outcomes."10 


20Light,  R.J.  (1990)  .  The  Harvard  assessment  seminars: 
Explorations  with  students  and  faculty  about  teaching,  learning  and 
student  life.   Cambridge,  MA:  Harvard  University. 
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The  Division  of  Disadvantaged  Assistance  (DDA)  should  encourage  HCOP  applicants 
to  incorporate  collaborative  learning  approaches  in  their  respective  curricula.  The 
research  has  shown  statistically  significant  proof  that  collaborative  learning  approaches 
such  as  Supplemental  Instruction  (SI)  and  Video-Based  Supplemental  Instruction  (VSI) 
can  significantly  improve  the  odds  of  students  whose  grade  point  average  and /or 
standardized  test  scores  place  them  in  a  relatively  high-risk  category  for  completing 
health  professions  curriculum.  Grantees  offering  post  baccalaureate  instruction  designed 
to  prepare  students  for  success  in  health  professions  schools  will  find  SI  and  VSI  helpful 
adjuncts  to  their  curricula.  As  educators  extend  their  reach  into  the  high  schools  and 
junior  high  schools  to  create  pathways  that  will  fulfill  the  goals  of  the  Association  of 
American  Medical  Colleges'(AAMC)  Project  "3000  by  2000",  they  should  introduce 
collaborative  learning  modes  of  instruction  in  their  programs  as  early  as  possible.  While 
the  AAMC  Project  focuses  on  gaining  future  medical  students,  its  concept  has  provided 
the  foundation  and  motivation  for  programs  in  all  areas  of  the  health  and  science 
professions.  The  recommendations  listed  below,  together  with  their  corresponding 
guidelines  (detailed  in  Chapter  VI),  will  facilitate  HCOP-wide  goals  and  the 
implementation  of  effective  educational  interventions  approaches. 


RECOMMENDATIONS 

The  following  recommendations  have  an  excellent  chance  of  being  well  received 
by  HCOP  grantee  applicants  because  they  place  emphasis  on  approaches  either 
currently  being  used  by  some  HCOP  grantees  or  methodologies  originally 
developed  with  HCOP  financial  support. 

#1  Encourage  grantees  to  implement  learning  approaches  that  have  been 
shown  to  be  statistically  effective  in  improving  student  comprehension  and 
conceptual  understanding  of  lectures  and  textbook  material. 

#2  Encourage  all  grantees  to  become  familiar  with  collaborative  learning 
approaches;  and,  to  adopt  those  approaches  which  can  best  be  supported 
by  their  existing  resources  and  are  most  appropriate  for  the  students 
targeted  by  their  program. 

#3  DDA  should  minimally  earmark  funds  to  support  at  least  one 
Grantee  in  each  region  that  has  its  objective  the  improvement  of  the 
science  and  math  portions  of  standardized  tests;  and,  makes  a 
commitment  to  use  collaborative  learning  approaches  to  improve 
basic  arithmetic  operations. 
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#4  Grantees  should  be  encouraged  to  sponsor  reading  interventions 
that  emphasize  evaluation,  application,  and  the  incorporation  of 
new  information. 

#5  Grantees  should  be  encouraged  to  sponsor  writing  interventions  that 
emphasize  addressing  the  challenges  that  have  been  incorporated  in 
the  writing  section  of  the  MCAT  (i.e.  to  define  the  given  statement 
as  thoroughly  as  possible,  to  explore  the  meaning  of  the  given 
statement  by  giving  a  contradictory  example  and  to  discuss  ways  to 
resolve  the  conflict  between  the  initial  statement  and  the 
contradictory  example). 


#6  Grantees  should  be  encouraged  to  implement  activities  in  HCOP  programs 
to  cultivate  non-cognitive  skills,  such  as  realistic  self-appraisal  and  positive 
self  concept. 

#7  Establish  a  process  by  which  HCOP  program  participants  will  leave 
the  program  with  diagnostic  results,  specific  information  about  the 
programs  learning  approaches,  outcome  performance  and 
recommendations  designed  to  enhance  future  development  of 
cognitive  skills. 
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CHAPTER  III 
SELECTION  OF  THE  HCOP  PARTICIPANT 


Chapter  III  reports  on  the  selection  of  participants  for  HCOP  funded  enrichment 
activities.  As  outlined  in  the  introduction,  this  portion  of  the  report  meets  the 
deliverable  as  directed  under  Task  6  of  the  project  entitled  "Research  on 
Minority /Disadvantaged  Support  Programs."  A  specific  objective  of  Task  6  is  to 
"...develop  and  test  criteria  for  identifying  the  characteristics  of  minority  students  with 
marginal  academic  credentials  who  are  most  likely  to  improve  their  competitiveness  for 
acceptance  into  and  retention  in  health  professions  training  programs  by  participating 
in  Federal  support  programs  such  as  the  Health  Careers  Opportunity  Program  (HCOP)". 

The  importance  of  defining  the  characteristics  of  the  "marginal"  or  "at  risk"  HCOP 
student  participant  can  best  be  appreciated  in  the  context  of  the  need  of  the  Division  of 
Disadvantaged  Assistance  (DDA)  to  more  efficiently  allocate  scarce  resources.  Political, 
legal  and  financial  forces  are  converging  in  a  manner  that  has  made  the  process  for 
identifying  "marginal"  students  and  determining  their  level  of  'relative  risk,'  critical.  In 
today's  fiscally  conservative  times,  programs  have  little  incentive  to  concentrate  limited 
resources  on  minority /disadvantaged  students  who  are  unlikely,  even  with  support 
services,  to  be  accepted  into  and  subsequently  complete  the  course  of  study  necessary 
to  graduate  from  a  health  professions  school. 

This  chapter  is  the  result  of  an  extensive  literature  search,  the  review  of  44  specially 
selected  HCOP  Progress  Reports  (reflecting  the  activities  of  10  individual  projects),  and 
the  deliberations  of  a  HCOP  Work  Group  (HWG)  comprised  of  experienced  educators 
who  have,  or  have  had,  HCOP  grants.  From  these  sources,  several  challenges  were 
faced  in  the  consideration  of  selection  criteria  for  HCOP  participants: 

•  The  terminology  commonly  used  to  categorize  prospective  HCOP  students, 

such  as  "at  risk",  "at  a'cademic  risk"  or  "marginal"  tend  to  be  poorly 
defined.  The  definition  of  the  "at  risk"  student  in  the  HCOP  Resource 
Book  (September,  1994  HCOP  Technical  Workshop,'  Rockville,  Md.)  is  an 
excellent  example  of  this  point.  The  HCOP  Resource  Book  quotes  "at  risk" 
as:  "low  science  GPA  scores;  low  admission  test  scores;  age  and /or 
prolonged  absence  from  academic  pursuits;  non-traditional  majors;  and, 
prior  preparation  at  an  institution  with  a  less  than  substantial  academic 
[curriculum]."  Unfortunately,  this  definition  does  not  translate  into  a 
mechanism  useful  in  applying  these  variables  in  a  way  that  takes  into 
consideration  the  differences  in  the  characteristics  of  these  variables 
exhibited  by  applicants  --  eg.  the  grade  point  average  (variable)  of  a 
student  may  differ  based  upon  the  competitiveness  of  the  school 
(characteristic). 
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The  process  of  categorizing  minority  students  as  "risk"  or  "marginal"  often 
results  in  the  misrepresentation  that  these  categories  define  all  minority 
students.  It  is  safe  to  conclude,  using  any  measure,  that  minority  students 
are  not  all  "marginal"  or  "at  risk."  Unfortunately,  an  unacceptable  large 
number  of  minority  students  do  have  one  or  more  of  the  "risk  factors" 
described  in  the  HCOP  Resource  Book.  It  is  with  the  goal  of  serving  the 
students  in  this  latter  group,  that  this  report  is  addressed. 

The  literature  addressing  the  identification  of  "risk"  or  "marginal"  student 
characteristics  has  largely  involved  only  pre-medical  and  medical  students. 
An  objective  of  the  contract  is  to  identify  characteristics /variables/ 
indicators  that  could  be  broadly  acceptable  to  HCOP  grantees.  The  HWG 
recommended  establishing  a  model  that  could  be  adapted  to  all  health 
professions. 

The  fact  that  Cognitive  and  Non-Cognitive  Attributes  are  useful  in 
measuring  the  level  of  "relative  academic  risk"  related  to  a 
minority/disadvantaged  student's  acceptance  into,  and  graduation  in  a 
timely  manner  from,  a  health  professions  school.  The  HWG  pointed  out 
that  the  Association  of  American  Medical  Colleges  (AAMC)  has  already 
developed  an  extensive  database  on  student  cognitive  and  non-cognitive 
profiles  representing  medical  schools.  Data  derived  from  various  sources, 
including  the  AAMC's  American  Medical  College  Application  Service 
(AMCAS)  database  and  individual  medical  schools,  are  constantly  analyzed 
and  have  led  to  volumes  of  medical  education  literature.  This  information 
is  commonly  assessed  to  identify  the  relationship  between  student 
undergraduate  cognitive  characteristics  to  academic  performance  in 
medical  school.  However,  only  a  small  portion  of  the  literature  published 
from  the  AAMC  database  has  been  devoted  to  factors  that  reflect  how 
minority/disadvantaged  students  perform  in  medical  school  courses. 

The  number  of  variables  and  their  associated  characteristics  that  could 
contribute  to  the  "academic  risk"  status  of  a  student  are  numerous. 
Therefore,  it  is  not  the  best  use  of  limited  resources  to  focus  principally  on 
defining  characteristics  that  would  be  "universally"  accepted  by  HCOP 
grantees.  However,  it  should  be  possible  to  identify  .a  number  of  dominate 
variables  to  which  each  program  would  have  the  option  of  evaluating 
these  variables  in  the  context  of  a  range  of  characteristics.  The  HWG 
members  were  confident  that,  in  a  relative  sense,  the  same  dominate 
variables  that  contribute  to  minority/disadvantaged  medical  students  being 
at  "academic  risk"  would  also  constitute  risk  factors  in  other  health 
professions. 
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In  the  context  of  these  issues,  Chapter  III  will  address: 

•  Cognitive  and  Non-Cognitive  Variables  and  Their  Relationship  to  the 
Indicators  of  "Relative  Academic  Risk," 

•  Developing  Tools  for  Measuring  Relative  Risk  Levels. 

•  Recommendations  for  the  Selection  of  Students  likely  to  benefit  from 
participation  in  HCOP  interventions 


COGNITIVE  AND  NON-COGNITIVE  VARIABLES  AND  THEIR  RELATIONSHIP 
TO  THE  INDICATORS  OF  'RELATIVE  ACADEMIC  RISK' 

The  Role  of  Cognitive  and  Non-Cognitive  Variables  in  Predicting  Academic 
Performance: 

The  terms  and  definitions  presented  in  the  following  sections  of  this  analysis  were 
derived  from  the  literature  together  with  HWG  recommendations  and  the  review  of 
program  reports. 

The  identification  and  enumeration  of  Cognitive  and  Non-Cognitive  variables  helps  to 
define  student  attributes  and  skills  into  measurable  data  units  which  can  be  assigned 
weights,  imputed  into  an  algorithm  and  used  to  predict  of  academic  performance.  The 
research  shows  that  when  variables  are  given  quantifiable  values  as  a  result  of  their 
input  into  an  analysis  tool,  such  as  the  Selection  Index  detailed  later  in  this  chapter,  the 
result  is  a  student  profile  indicating  relative  academic  risk.  The  comparability  of  each 
variable  is  dependent  on  many  factors.  "Relative  levels  of  academic  risk"  should  be 
measured  when  risk  is  considered  in  the  context  of  a  student's  potential  for  ultimate 
acceptance  into,  and  completion  of,  the  course  of  study  necessary  to  graduate  from  a 
health  professions  school. 

For  the  purpose  of  this  analysis,  characteristics,  variables  and  their  corresponding 
indicators  will  be  grouped  into  one  of  two  categories  —  Cognitive  and  Non-Cognitive, 
the  configurations  of  which  are  described  as: 


Cognitive  Attributes  are  those  abilities  and  skills  which  are  utilized  to  develop 
independent  thought  and  to  deduce  answers  for  multi-facetted  questions. 

Non-Cognitive  Attributes  are  those  abilities  and  skills  which  aid  individuals  to 
operate  more  effectively  within  society,  such  as,  ability  to  concentrate  and  be 
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persistent  in  completing  tasks,  willingness  to  work  hard,  ability  to  identify  clear 
and  realistic  goals  and  objectives. 


Within  this  context: 

•  "Characteristics"  are  the  qualities  that  distinguish  a  given  variable  -  eg 
institutional  competitiveness  can  be  used  to  distinguish  grade  point  averages. 

•  "Variables"  are  the  units  for  input  into  selection  indexes  in  order  to  measure 
relative  student  risk  levels, 

•  "Indicators"  are  defined  as  those  'flags'  or  'markers'  which  evolve  from  the  input 
of  variables  into  a  selection  index,  or  other  measures,  and  can  be  used  by 
educators  and  program  directors  to  identify  which  potential  HCOP  participants 
are  most  likely  to  succeed  in  the  program. 


Identifying  Variables  and  Establishing  a  Basis  for  Their  Relative  Weights: 

Program  experience  coupled  with  correlation  analysis  can  produce  "reasonable" 
weighting  of  variables.  This  initial  process  takes  into  account  the  unique  value  a 
program  might  assign  to  the  characteristics  of  each  variable.  The  subsequent  validation 
of  the  importance  of  the  variable  and  its  associated  characteristics  can  be  accomplished 
through  step-wise  regression  analysis. 

One  of  the  key  challenges  facing  HCOP  grantees  is  the  vast  number  of  possible 
characteristics,  variables  and  resultant  indicators,  which  could  contribute  to  the 
measurement  of  "relative  academic  risk."  Equally  as  important,  what  may  be  considered 
a  basis  for  risk  by  one  grantee  may  not  be  a  risk  for  another.  For  example,  commuting 
time  may  be  a  challenge  for  a  HCOP  program  participant  and /or  health  professions 
school  student  in  a  rural  or  suburban-area,  but  may  not  be  as  much  of  a  risk  factor  for 
residents  of  urban  areas  with  extensive  public  transportation  systems. 

Within  this  construct,  the  relative  level  of  "risk"  that  a  HCOP'  grantee  applies  to  a 
student  should  be  a  function  of  a  number  of  dominate  factors.  The  relative  value  of 
each  factor  should  be  defined  in  relation  to: 

•  program  goals  and  objectives, 

•  the  characteristics  and  needs  of  the  target  student  population, 

•  curriculum  design  and  available  educational  tools, 


27 


•  staffing  skills  and  knowledge  base,  and, 

•  program  financial  resources. 

Even  though  the  legislative  objectives  guiding  HCOP  projects  are  the  same  for  each, 
individual  projects  are  free  to  establish  additional,  personalized  goals  and  objectives.  As 
a  result,  objectives  and  activities  vary  greatly  from  project  to  project.  As  a  result, 
objectives,  the  make-up  of  target  student  populations,  curricula,  quality  and  quantity  of 
staff,  and  basic  financial  resources  all  have  some  influence  on  the  academic  performance 
of  HCOP  participants. 

The  matching  of  HCOP  participants  with  the  programs  most  likely  to  benefit  them 
requires  that  HCOP  grantees  be  able  to  predict  how  the  each  applicant's  cognitive  and 
non-cognitive  attributes  will  interplay  within  the  academic  and  non  academic  project 
environment.  The  research  indicates  that  any  assessment  of  variables  is  best 
accomplished  on  a  project-specific  basis,  against  measures /indices  developed  by  each 
HCOP  grantee. 

The  HWG  members  recommended  that  research  in  this  area  should  begin  with  the 
literature  which  correlates  medical  school  admission,  retention  and  graduation  with 
cognitive  and  non-cognitive  variables.  HWG  felt  very  strongly  that  while  this  research 
utilized  medical  and  pre-medical  school  student  data,  its  results  could  be  generalized  to 
nearly  all  HCOP  projects  and  other  health  professions.  The  results  of  these  analyses 
show  a  statistically  significant  correlation  between  cognitive  variables,  such  as  college 
overall  grade  point  average  and  the  MCAT  physical  science  and  biological  sub-test 
scores,  with  pre-clinical  or  basic  science  medical  school  course  grades. 

For  example,  one  such  study  published  by  Friedman  and  Bakewell21,  focused  on  the 
relationship  between  MCAT  sub-tests  and  the  performance  of  158  first-year  University 
of  North  Carolina  (class  of  1978)  medical  students  on  the  year-end  comprehensive 
examination.  They  discovered  that  combinations  of  GPA  and  MCAT  scores  accounted 
for  about  40%  of  achievement  variance.  These  results  were  confirmed  through  several 
studies  conducted  at  various  universities  throughout  the  United  States.  n  a  24 


21  Friedman,  Charles  and  Bakewell,  William. 
"Incremental  Validity  of  New  MCAT".  Journal  of 
Medical  Education.  Vol  55,  May,  1980. 

Cullen,  Thomas  and  Dohner,  Charles. 
"Predicting  First  Quarter  Test  Scores  from  the 
New  Medical  College  Admission  Test" .  Journal 
of  Medical  Education.  Vol  55,  May,  1980. 
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The  studies  demonstrate  that  it  is  possible  to  generally  predict  a  student's  relative  risk 
of  getting  poor  or  'marginal'  grades  in  the  first  and  second  years  of  medical  school  basic 
science  courses  with  some  accuracy.  However,  few  studies  isolated  minority-specific 
data  which  would  have  provided  insights  into  any  differentiation  between  the  student 
populations.  Additionally,  the  fact  that  a  portion  of  the  minority /disadvantaged 
students  with  modest  overall  GPAs  and  standardized  test  scores  who  are  accepted  into 
health  professions  schools  do  achieve  competitive  grades  indicates  that  other  factors, 
beyond  their  ability  to  score  well  on  undergraduate  academic  tests  and  course  work, 
play  a  role. 

The  need  for  this  information  led  Sedlacek  and  others  to  search  for  other  variables  that 
might  predict  the  academic  performance  of 

under  represented  minorities  (URM)  students.  Their  studies  indicated  that  relative 
academic  risk  could  be  better  estimated  by  considering  non-cognitive  attributes  (as  well 
as  cognitive  attributes)  such  as  realistic  self  appraisal  and  positive  self  within  the  context 
of  the  pool  of  weighted  variables.  In  this  manner,  non-cognitive  attributes  can  better 
help  identify  which  minority/disadvantaged  students  are  likely  to  remain  in  school  and 
graduate  in  a  timely  manner,  than  can  be  accomplished  through  the  consideration  of 
grade  point  average  or  standardized  test  scores  (cognitive  variables)  alone. 


Key  Variables  and  Indicators  for  Predicting  URM  Academic  Performance: 

The  literature  shows  that  preclinical  medical  school  academic  performance  by  "majority 
students"  can  be  predicted  through  the  analysis  of  cognitive  variables  alone.  This  is 
generally  true  also  in  the  case  of  students  who  rank  in  the  25  percentile  of  the  nation.25 
However,  research  focusing  on  the  spectrum  of  attributes  exhibited  by  "marginal" 
minority/disadvantaged  students  begs  further  consideration  of  a  broader  range  of 
variables. 


23  McGuire,  F.L.  "The  New  MCAT  and  Medical 
Student  Performance".  Journal  of  Medical 
Education.  Vol  55,  1980. 

Brooks,  C.  Michael  and  Jackson,  James. 
"Validity  of  the  New  MCAT  for  Predicting  GPA 
and  NBME   Part   I   Examination   Performance" . 

Journal   of   Medical   Education.   Vol   56, 
September,  1981. 

25  Anderson,  J.  and  Mitchell,  K.  (1986)  .  Pre- 
Admission  Data  and  Medical  Performance  for 
Black  Medical  Students,  from  paper  presented 
at  AAMC  Conference. 
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Sedlacek,  Prieto  and  other  educational  researchers  assert  that  cognitive  variables  alone 
(grade  point  average,  standardized  tests,  course  grades)  are  not  as  effective  as  predictors 
of  academic  success  (achievement  or  persistence  in  a  curriculum  of  study)  for  under 
represented  minority  (URM)  students  as  they  are  for  majority  students.26  Although 
Sedlacek  has  primarily  studied  non-health  professions  students,  his  research  does  point 
to  the  fact  that  non-cognitive  attributes  —  such  as  realistic  self-appraisal  and  positive  self- 
concept—  taken  together  with  selective  cognitive  variables  (such  as  GPA  and  SAT)  can 
be  used  to  help  predict  students'  potentials  for  success  in  various  academic  curricula. 

Sternberg27  explains  the  difference  between  the  predictive  factors,  as  they  relate  to  URM 
versus  "majority  students,"  by  focusing  on  the  student's  societal  experiences,  and  how 
these  experiences  influence  patterns  of  learning.  Sedlacek  and  Sternberg  suggest  that 
minority  students  develop  and  foster  learning  behaviors  to  deal  with  different  societal 
challenges,  such  as  cultural  barriers,  which  majority  students  have  not  commonly 
experienced.  It  is  these  additional  adaptive  strategies  which  can  contribute  to  a 
minority /disadvantaged  student's  ability  to  succeed  both  in  the  university  environment 
and  as  a  capable  health  professional. 

Surprisingly  to  some,  studies  conducted  at  predominately  minority  universities  such  as 
Howard28  and  Xavier29  conclude  that  GPA's  and  MCAT  subtest  scores  are  strong 
predictors  of  academic  performance  for  minority  students  enrolled  at  their  respective 
institutions.  This  conclusion  is  understandable  in  light  of  the  fact  that  the  students 
evaluated  in  these  studies,  and  others  like  them,  have  largely  homogeneous  academic 
profiles  (primarily  composed  of  high  GPAs  and  standardized  test  scores)  which  is  in 
direct  parallel  to  the  circumstance  within  competitive  majority  schools. 
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DEVELOPING  TOOLS  FOR  MEASURING  RELATIVE  RISK  LEVELS 

Justification  for  the  Development  of  Individualized  Selection  Indices: 

Sedlacek  and  Webb30  are  currently  involved  in  research  which  may  lead  to  the  listing 
of  additional  non-cognitive  variables  useful  in  predicting  medical  school  performance. 
While  the  work  of  Sedlacek,  Webb,  Prieto,  Sternberg  and  others  provides  a  foundation 
for  Selection  Indices  that  incorporate  non-cognitive  variables,  the  assessment  of  cognitive 
skills  alone  has  been  proven  to  provide  a  'jumping-off  point'  for  further  analysis.  For 
example,  Burke  at  the  University  of  Illinois  College  of  Medicine  at  Urbana-Champaign 
showed  that  12  URM  students  who  faced  either  dismissal  or  a  repetition  of  the  year 
(1983-1984),  11  were  pre-identifiable  on  the  basis  of  the  Science  GPA,  Chemistry  MCAT, 
and  Physics  MCAT,  a  'true-positive'  rate  of  about  92%.  These  findings  were  important 
for  the  University  of  Illinois  College  of  Medicine  at  Urbana-Champaign-  They  pointed 
to  a  target  group  of  "at  academic  risk"  students  who  could  have  been  identified  and 
provided  academic  support  services  early  in  the  academic  process. 

These  findings,  while  insufficient  to  draw  a  complete  picture  of  each  student's  relative 
risk  have  encouraged  several  health  professions  school  to  perform  an  in-depth  analysis 
of  the  relationship  of  cognitive  and  non-cognitive  variables  with  the  academic 
performance  of  enrolled  URM  students.  The  application  of  an  extensive  step-wise 
regression  analysis  of  dependent  and  independent  variables  is  key  to  generating  a 
selection  index  that  defines  relative  academic  risk.  A  selection  index  may  enable 
admissions  committees  to  limit  their  selection  of  students  at  high  "academic  risk"  to 
those  who  have  at  least  a  50%  chance  of  graduating  from  the  health  professions  school 
in  a  timely  manner. 

However,  grantees  should  also  be  cautioned  about  the  pitfalls  of  placing  too  much 
weight,  and  /or,  collecting  highly  personal  information  which  could  be 
disproportionately  weighted  in  comparison  to  its  true  value  in  predicting  a  student's 
academic  performance.  Institutions  need  to  be  careful  when  utilizing  (and 
communicating  the  results  of)  non-cognitive  variables  whose  measurements  are  highly 
subjective. 

Appendix  4  identifies  the  variables  and  regression  formulae  that' are  currently  used  by 
a  nationally  recognized  medical  school  to  calculate  the  Selection  Index  for  its  College  of 
Medicine.  It  should  be  noted  that  its  Selection  Index  consists  of  both  a  Cognitive  Index 
and  a  Non-Cognitive  'Diversity'  Index. 


■- 


Webb,  Carmen  M.D. ,  Assistant  Professor  and  Researcher  for 
the  Office  of  Medical  Education,  Medical  College  of 
Pennsylvania  and  Hahnemann  University. 
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It  is  also  important  to  point  out  that  use  of  the  selection  index  has: 

•  increased  faculty  confidence  in  the  decision-making  of  the 
admissions  committee;  which  in  turn,  buttressed  faculty 
support  for  continued  minority  student  admissions;  and, 

•  enabled  the  admissions  committee  to  select  URM  students 
with  modest  academic  profiles  who  were  at  moderate-to-low 
risk  of  not  completing  their  medical  education. 

It  is  likely  that  all  institutions  of  higher  education  use  some  type  of  "indexing"  in  the 
selection  of  their  respective  students.  However,  the  majority  of  institutions  do  not  have 
"rigid"  indices  based  on  statistically  valid  formulae.  It  is  often  the  case  that  admission 
committees  of  these  institutions  make  acceptance  and  rejection  decisions,  particularly 
those  involving  URM  students,  on  the  basis  of  "generally  understood  cognitive  and  non- 
cognitive  cut-off  variables."  For  example,  it  is  frequently  the  case  that  admissions 
committees  reject  any  student  with  a  MCAT  sub-test  score  of  4  or  less,  and /or  any 
student  who  had  a  GPA  of  less  than  2.30. 

Dependence  upon  Standardized  Tests,  such  as  the  MCAT,  as  a  Principle  Basis  for 
Selection: 

The  suspicion  of  grade  inflation  and  unfamiliarity  with  historically  Black  colleges  or 
smaller /less  known  undergraduate  schools,  encourages  health  professions  school 
admission  committee  members  to  view  national  standardized  test  scores  as  one  of  the 
"fairest"  parameters  in  the  selection  of  URM  students.  Unfortunately,  this  line  of 
reasoning  too  often  provides  admission  officers  and  admission  committees  an  "easy  out" 
in  the  selection  process.  In  fact,  logic  indicates  that  there  is  a  certain  moral  comfort  in 
placing  the  difficult  decision  of  acceptance  or  rejection  on  the  shoulders  of  an  "objective 
test"  such  as  the  MCAT. 

This  point  was  underscored  during  the  HCOP  Work  Group  (HWG)  sessions.  One  of  the 
members  summarized  an  unpublished  study  which  indicated  that  one  hundred  and 
fourteen  (114)  medical  school  admission  officers  ranked  their  selection  variables  (once 
initial  screening  has  been  completed)  in  the  following  order  of  importance: 

•  medical  school  interviews  and  reference  letters,  are  weighted 
highest,  followed  by  - 

•  undergraduate,  post-baccalaureate  and /or  graduate 
cumulative  or  science  GPA  average 

•  MCAT  score 
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Most  of  the  HWG  members  did  not  find  the  ranking  priorities  surprising,  given  the  fact 
that  the  majority  of  enrolled  medical  students  have  relatively  similar,  high  grade  point 
averages  and  high  MCAT  scores.  Nearly  all  students  who  are  finally  considered  for 
acceptance  by  a  medical  school  admissions  committees  have  science  and  overall  grade 
point  averages  that  fall  with  in  the  3.5  to  4.0  range.  That  same  group  of  students  have 
MCAT  sub-test  scores  that  fall  in  the  range  of  9-15  and  total  MCAT  scores  that  fall  in 
the  range  27-42. 31  Therefore,  the  interviews  and  reference  letters  are  the  mostly  likely 
points  of  differentiation  during  the  selection  process.  The  homogeneity  among  majority 
students  is  related  to  the  fact  that  students  with  lower  scores  and  GPA's  are  not  willing 
to  spend  significant  resources  on  applications  which  are  highly  unlikely  to  be  successful. 


While  the  study  results  did  not  surprise  any  of  the  HWG  members,  they  were 
unanimous  in  their  concern  about  how  the  results  tend  to  be  used  and  interpreted  by 
the  public.  It  is  their  belief  that  contrary  to  the  final  selection  process  outlined  above, 
medical  schools  initially  focus  on  the  selection  variables  which  are  applied  to  URM 
student  applications  in  the  following  order: 

•  MCAT  scores 

•  undergraduate,   post-baccalaureate   and/or  graduate   GPA 
averages 

•  medical  school  interviews  and  letters  of  reference 

A  probable  explanation  for  this  behavior  stems  from  the  fact  that  URM  medical  student 
profiles,  with  regards  to  their  grade  point  averages  and  MCAT  scores,  are  much  more 
diverse  than  those  of  majority  medical  students.  A  large  percentage  of  minority  students 
who  are  finally  considered  for  acceptance  by  a  majority  medical  school  admissions 
committees  have  a  science  and  overall  grade  point  average  that  falls  in  the  range  of  2.3 
to  3.5.  That  same  group  of  students  have  MCAT  sub-test  scores  that  fall  in  the  range 
of  5-8  and  total  MCAT  scores  that  fall  in  the  range  16-26.  It  should  be  pointed  out  that 
minority  students  in  this  "acceptable  pool,"  often  have  high  grade  point  averages  G>3.4), 
but  low  MCAT  scores;  or  often  have  high  MCAT  scores  (>  8) -and  low  grade  point 
averages  (<  2.5).  Relatively  few  have  both,  high  grade  point  averages  and  high  MCAT 
sub-test  scores. 

The  HCOP  Work  Group  (HWG)  was  unanimous  in  its  belief  that  including  variables  such  as 
an  applicant's  participation  in  a  HCOP  or  HCOP-type  program  into  the  admissions  process 
could  reduce  concern  about  a  student's  relative  risk  level  based  on  MCAT  scores  alone. 


31     1994  Association  of  American  Medical  Colleges' 
MCAT  Reporting  Series . 
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Finally,  at  the  same  time  that  HCOP  grantees  work  to  change  the  criteria  by  which  URM 
students  are  selected,  it  is  necessary  to  help  the  present  URM  students  to  meet  the 
selection  requirements  of  health  professions  admissions  committees.  This  means 
improved  MCAT  scores,  improved  GPA,  and  improved  interviews  as  addressed  in 
Chapter  II  of  this  report. 


Justification  for  Consideration  of  Other  Variables: 

Published  correlation  studies  report  that  the  Physical  Science  MCAT  sub-test  score  can 
explain  30  to  40  percent  of  the  variance  in  medical  school  basic  science  course  grades. 
Grade  point  averages  also  evidence  a  higher  correlation  to  medical  school  basic  science 
course  performance.  However,  these  statistical  correlations  primarily  depict  the 
academic  performance  of  "majority  medical  students."32 

One  key  variable  which  would  help  predict  the  performance  of  URM  students  — 
previous  HCOP  participation  —  rarely  is  reported.  Nearly  all  statistical  correlation 
models,  focusing  on  URM  academic  performance,  include  relatively  little  or  incomplete 
information  on  the  student's  previous  participation  in  one  or  more  HCOP  or  HCOP-like 
program(s).  The  studies  that  have  correlated  participation  in  an  HCOP  program  with 
health  science  school  performance: 

•  do  not  include  indications  about  whether  any  effort  had  been 
made  to  see  if  the  student  had  participated  in  more  than  one 
HCOP  or  HCOP-like  experience; 

•  do  not  provide  data  assessing  the  relative  impact  of  the 
HCOP  experience  on  the  participant's  course  performance  in 
comparison  to  cognitive  and  other  non-cognitive  variables. 

The  success  of  HCOP  postbaccalaureate  programs  (see  appendix  6)  strongly  suggests 
that  participation  in  a  HCOP  or  HCOP-like  academic  eru-ichment  experience  has  reduced 
the  relative  risk  of  selective  URM  students.  There  is  evidence  that  the  reduction  in  risk 
has  been  accomplished  with  only  a  modest  improvement  in  the  participant's  MCAT 
subtest  scores.  As  illustrated  in  Appendix  4,  there  is  a  need  to  utilize  step-wise 
regression  formulae  to  statistically  predict  if  previous  HCOP  or  HCOP-like  post 
baccalaureate  program  participation  will  reduce  the  relative  risk  of  URM  students  with 
marginal  academic  profiles. 


Association  of  American  Medical  Colleges.   (1994)  MCAT 
Validity  Studies.  Washington,  DC 
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FINDINGS  AND  RECOMMENDATIONS  FOR  THE  SELECTION  OF  STUDENTS 
LIKELY  TO  BENEFIT  FROM  PARTICIPATION  IN  HCOP  INTERVENTIONS. 

Findings  on  Participant  Selection  Process: 

The  literature  is  filled  with  studies  which  show  that  undergraduate  cumulative  grade 
point  averages  (UCGPA)  and  undergraduate  cumulative  science  GPAs  (UCSGPA), 
together  with  physical  and  biological  science  MCAT  sub-test  scores,  have  statistically 
significant  correlations  to  successful  performance  in  medical  school  basic  science  courses. 


Within  the  context  of  minority  /disadvantaged  students,  the  HCOP  Progress  Reports,  as 
well  as,  the  comments  expressed  by  the  HCOP  Work  Group  (HWG)  participants, 
support  the  philosophy  that  non-cognitive  attributes  should  be  an  important  part  of  the 
academic  risk  assessment.  A  number  of  HCOP  programs  have  collected  non-cognitive 
data  utilizing  a  validated  diagnostic  test;  but,  only  a  few  have  analyzed  the  non- 
cognitive  data  to  predict  levels  of  academic  risk. 

The  key  to  the  selection  of  minority/disadvantaged  students  who  are  more  likely  to 
benefit  from  HCOP  interventions,  is  the  application  of  both  cognitive  and  non-cognitive 
criteria  within  a  structured  and  controlled  selection  index  process.  This  will  yield 
greater  insights  into  the  capabilities  of  individual  applicants  and  a  comprehensive  profile 
of  the  indicators  which  are  most  useful  in  determining  a  student's  level  of  academic  risk. 


The  findings  outlined  in  this  report  describe  the  role  of  many  cognitive  and  noncognitive 
variables  which  should  be  considered  when  assessing  a  student's  academic  risk. 
Unfortunately,  terms  such  as  "academic  risk"  and  "marginal"  student  are  not  clearly 
defined.  Variations  between  programs  show  different  meanings  from  one  HCOP/health 
professions  admissions  committee  to  another.  This  report  underscores  the  fact  that 
"relative  risk"  and  "marginality"  tend  to  be  defined  primarily  by  cognitive  variables 
within  the  context  of  institutional  or  program  biases.33 

The  magnitude  and  variety  of  the  factors  that  come  to  bear  -on  the  prediction  of 
"academic  risk"  suggests  it  would  be  fruitless  to  try  to  develop  a 'common  definition  of 
"academic  risk"  that  would  be  "universally"  acceptable  to  HCOP  programs  and /or  to 
health  professions  schools.    Therefore,  this  report  proposes  that  each  HCOP  program 


33 


Highlights    of    On-Site    Interview   Findings, 
Evaluation   of   the   Levels   of   Academic   Risk 
Associated  with  Applicants   to   HCOP   supported 
Health   Professions   Schools,   March   22,   1991, 
prepared  under  contract  #240-90-0034,  submitted  by 
Levine  Associates,  Inc. 
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attempt  to  quantify  levels  of  student  risk  based  on  the  following  indicators: 

•  the  academic  and  non-academic  profiles  of  the  student  applicants;  and, 

•  the  program  performance  objectives;  and, 

•  the  financial  and  non-financial  resources  available  to  support  the  HCOP  program. 


The  following  recommendations  reflect  these  findings  and  the  need  for  greater 
information  to  support  the  process  of  HCOP  participant  selection.  Each  of  the  targeted 
recommendations  included  below  is  designed  to  enhance  the  capacities  of 
minority /disadvantaged  students  to  gain  competitive  acceptance  into  and  timely 
graduation  from  the  health  professions  schools. 


Recommendations 

#1  Encourage  HCOP  Grantees  to  utilize  both  cognitive  and  non-cognitive  variables 
as  a  basis  for  the  selection  of  HCOP  participants  and  the  matching  of  those 
participants  to  the  programs  which  are  most  likely  to  benefit  them. 

#2  Grantees  should  be  encouraged  to  collect  the  "minimum  data  set"  (discussed 
in  Chapter  IV)  together  with  additional  information  from  student 
applications,  letters  of  recommendation,  by  telephone  or  personal  interview 
and  when  possible  by  means  of  pre-admission  cognitive  and  non-cognitive 
assessment  tests. 

#3  While  Grantees  should  be  encouraged  to  use  non-cognitive  variables  in 
conjunction  with  cognitive  variables,  they  should  be  strongly  cautioned  on  how 
best  to  apply  this  information  within  the  emotionally  charged  context  of  the 
student  selection  process. 
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CHAPTER  IV 
ASSESSMENT  OF  HCOP  PROJECT  DATA 


If  Chapters  II  (Interventions)  and  III  (Participant  Selection)  can  be  thought  of  as  a 
description  of  the  'vehicle'  which  drives  HCOP  projects,  then  Chapter  TV  endeavors  to 
provide  the  'fuel'  which  powers  this  vehicle.  The  literature,  together  with  a  detailed 
analysis  of  program  progress  reports  and  recommendations  from  the  HCOP  Work 
Group  (HWG),  supports  the  premise  that  the  'fuel'  upon  which  quality,  cost-effective 
and  well  monitored  programs  must  run,  is  DATA. 

This  chapter  addresses  the  identification,  collection,  analysis  and  application  of  key  data 
elements  or  'variables'  as  they  relate  to  the  design  of  individual  HCOP  interventions,  the 
selection  of  HCOP  participants,  the  subsequent  monitoring  of  program  components  and 
the  assessment  of  program  impact,  particularly  over  time.  As  outlined  in  the 
Introduction  section  of  this  report,  Chapter  IV  relates  directly  to  the  deliverables  as 
defined  under  Tasks  6,  10  and  11  of  the  project  entitled  "Research  on 
Minority /Disadvantaged  Support  Programs."  These  tasks  encompass  the  development 
and  application  of  a  methodology  for  assessing  HCOP  Project  Reports  and  program 
outcomes.  In  conjunction  with  Task  #6,  this  report  will  provide  insights  into  linkages 
between  various  selection  criteria   and  HCOP  participant  progress. 

As  a  foundation  for  the  recommendations  included  at  the  conclusion  of  this  chapter,  the 
following  components  will  be  presented  within  this  section  of  the  report: 

•  Synopsis  of  Current  Data  Collection  Activities  as  Directed  by  the  Division  of 
Disadvantaged  Assistance, 

•  The  Effective  Organization  of  Data,  and, 

•  Findings   from   the  Review   of  Data   Encompassed   in  HCOP   Grantee 
Progress  Reports, 

►  Issues  and  Barriers  faced  by  HCOP  Projects  in  Collecting  Useful  Data, 

►  The  Importance  of  Collecting  Comparable  Data  (across  programs), 

•  Recommendations  for  Future  Data  Collections  Efforts. 
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SYNOPSIS  OF  CURRENT  DATA  COLLECTION  ACTIVITIES  AS  OUTLINED  BY 
THE  DIVISION  OF  DISADVANTAGED  ASSISTANCE 

Historically,  the  data  collection  activities  carried  out  by  HCOP  grantees  have  emphasized 
the  racial /ethnic  and  financial  profile  of  the  student  population,  the  number  of 
minority/disadvantaged  students  entering  health  professions  school,  and  the  retention 
and  graduation  of  minority/disadvantaged  students  from  health  professions  schools. 
The  structure  of  these  activities  is  grounded  upon  the  purposes  established  in  the  HCOP 
legislation  (recruitment,  preliminary  education,  facilitating  minority/disadvantaged 
student  entry  and  retention  within  health  professions  schools).  The  data  variables 
requested  are  identified  in  the  HCOP  grant  application  and  include  the  following: 

•  First  Year  Enrollment  of  Disadvantaged  Students  by  Race /Ethnicity  and  Year 

•  Total  School  Enrollment  by  Race /Ethnicity  and  Year 

•  Academic     Progression:     Graduating     Class    of    Disadvantaged     Students    by 
Race /Ethnicity  and  Year 

•  The  Number  of  Disadvantaged  Students  Enrolled  in  Health  Professions  Schools  and 
Allied  Health  Programs  -  listed  by  Academic  Year 

•  HCOP  Competitive  Application  Grids  are  organized  into  three  data  collection 
groupings: 

Total  Trainees 

-  from  metropolitan  areas 

-  from  non-metropolitan  areas 

-  from  areas  unknown 

-  above  low-income  level 

-  below  low-income  level 

-  income  level  unknown 

Recruitment  Activity  Beneficiaries 

-  total  individuals  provided  health  careers  information 

-  total  individuals  provided  health  careers  counseling 

-  total  individuals  provided  health  careers  lectures 

Current  and  Previous  Program  Participants  Tracked  During  Reporting  Period 

Grantees  provide  HCOP  with  the  data  outlined  above,  to  the  best  of  their  ability,  in  an 
"aggregate"  format,  utilizing  the  grids  provided  in  the  HCOP  Application  Guidance.  In 
addition,  supplemental  data  are  collected  on  a  program-specific  basis  in  response  to 
individual  needs. 
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THE  EFFECTIVE  ORGANIZATION  OF  DATA 

This  research  has  shown  that  the  collection  of  student-specific,  cohort  and  aggregate  data 
which  is  reliable  and  valid  provides  a  useful  foundation  for  the  design,  implementation 
and  assessment  of  HCOP  efforts.  The  HWG  shared  this  view  and  expressed  great 
enthusiasm  about  the  prospect  of  participating  in  the  development  of  guidelines  through 
which  appropriate  data  could  be  collected  and  analyzed.  They  also  expressed  a 
perspective  that  is  consistent  with  the  literature  review  —  that  the  cornerstone  of  the  data 
collection  process  is  Student-Specific  Data.  From  this  initial  stage  of  collection,  data  can 
be  organized  and  analyzed  in  Cohorts  and  in  the  Aggregate.  While  it  is  possible  to 
build  up  to  aggregate  data  from  individual  student  academic  records,  it  is  only  rarely 
possible  to  accomplish  the  reverse.  Within  this  context  data  collection  and  assessment 
can  be  seen  as  stages  in  an  evolutionary  process, 


moving  from  -^Student-Specific  to  — >  Cohort  and  leading  to  — >  Aggregate 


In  understanding  the  applicability  of  project  data  to  future  programming  efforts,  these 
data  'stages'  might  be  viewed  as  part  of  a  continuum  of  analysis  leading  to  the 
development  of  viable  and  effective  recommendations.  The  foundation  of  this  process 
is  in  the  collection  of  comparable  data  elements  which  can  be  used  to  statisticaDy 
validate  relationships  between  interventions  and  outcomes.  Therefore,  the  stages  can  be 
conceptualized  as  follows: 

Student-specific  data  provides  the  greatest  insights  about  the  relationships  between 
cognitive /non-cognitive  variables  and  academic  performance.  These  insights  are 
critical  for  the  development  of  reliable  participant  selection  criteria.  This  data  is  also 
valuable  for  the  initial  stages  of  intervention  design  and  to  determine  program  impact 
on  project  participants.  Caution,  however,  must  be  taken  to  ensure  that  student- 
specific  data  is  kept  confidential.  For  example,  a  minimal  level  of  confidentiality  may 
be  achieved  through  the  deletion  of  names  and  social  security  numbers  from  files  and 
replacing  this  information  with,  for  example,  HCOP  identification  codes. 

Cohort  data  presents  information  in  groupings  by  like  variables  which  are  useful  in 
drawing  correlations  between  program  participant  skill  levels  and  the  application 
requirements  of  various  health  professions  schools.  Additionally,  cohort  data, 
consistently  captured  in  "like  data  elements",  lends  itself  to  constructive  comparative 
and  impact  analysis  over  the  life  span  of  individual  HCOP-funded  efforts.  This  level 
of  analysis  is  very  useful  in  determining  the  transferability  of  a  given  intervention 
strategy. 
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Aggregate  data  offers  summaries  of  important  program  elements  and  participant 
profiles  useful  in  communicating  information  on  a  broad  spectrum  of  HCOP- 
supported  strategies.  In  addition,  it  is  frequently  used  to  provide  comparable 
assessments  of  programmatic  impacts.  However,  this  level  of  data  rarely 
demonstrates  a  clear  cause-and-effect  relationship  between  program  interventions  and 
individual  student  performance  . 

Grantees  are  frequently  called  upon  to  develop  'snap  shot'  analyses  which  provide  a 
generalized  picture  of  program  activity.  Aggregate  and  Cohort  data  are  particularly 
useful  for  fulfilling  this  purpose.  For  example,  prior  to  an  upcoming  university 
promotions  committee  meeting,  the  Dean  of  a  school  might  request  that  the  HCOP 
project  director  provide  a  quick  performance  review  of  current  HCOP  participants. 
Because  student  progress  may  vary  greatly  from  survey  point  to  point,  caution  should 
be  taken  with  respect  to  the  predictive  reliability  of  these  'snap  shot'  analyses. 

In  comparison,  student-specific  data  or  even  cohort  data,  which  is  tracked  over  several 
program  periods  and  preferably  between  project  cycles  (eg;  each  year  or  semester), 
includes  information  which  is  verified,  is  more  likely  to  provide  a  better  understanding 
of  the  risk  factors  associated  with  minority  /disadvantaged  HCOP  participants  and  the 
relative  impact  of  the  HCOP  intervention  employed. 


FINDINGS  FROM  THE  REVIEW  OF  DATA  ENCOMPASSED  IN  HCOP  GRANTEE 
PROGRESS  REPORTS 

HCOP  grantees  collect  data  that  varies  greatly  in  terms  of  the  amount,  type,  quality  and 
utility.  Grantees  use  a  wide  variety  of  reporting  formats  to  describe  their  participants 
and  to  support  claims  pertaining  to  program  effectiveness  and  general  outcomes.  An 
extensive  review  of  HCOP  Grantee  Progress  Reports  and  observations  from  the  HCOP 
Work  Group  (HWG)  indicates  that  HCOP  grantees  have  attempted  to  collect  and  report 
project  data  in  a  manner  that  reflected  HCOP's  legislative  intent  —  to  increase  in  the 
aggregate,  the  number  of  minority/disadvantaged  students  entering  and  graduating 
from  health  professions  schools.  Moreover,  the  HWG  expressed  the  opinion  that 
Grantees'  abilities  to  collect,  assess  and  present  detailed  data  anaJyses  were  limited  by 
the  amount  of  dollar  resources  each  program  could  allocate  to  the  data  assessment 
process,  staff  knowledge  of  quantitative  analysis  techniques,  as  well  as,  the  program's 
access  to  HCOP  participants'  post-program  academic  records. 

The  following  subsections  describe  the  levels  of  data  historically  collected  and  assessed 
by  HCOP  Grantees,  as  reflected  in  their  progress  reports  and  HWG  discussion.  This 
information  is  presented  in  an  order  which  ranges  from  the  most  frequently  reported 
level  of  data  to  the  less  frequently  reported  level: 
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Aggregate  Data 

In  appears  that  the  majority  of  HCOP  grantees  have  historically  presented  data  in  the 
aggregate.  This  data  has  been  generally  useful.  For  example,  its  value  is  reflected  in 
illustrating  the  correlation  between  HCOP-funded  programs  and  the  dramatic  increase 
(between  1970-1975),  and  subsequent  plateauing,  of  large  numbers  of 
underrepresented  minority  (URM)  matriculating  and  graduating  health  professions 
students.  It  has  also  enabled  the  Division  of  Disadvantaged  Assistance  to  track  and 
assess  the  impact  of  appropriated  dollars.  The  outcome  information  generated  by  this 
aggregate  data  shows  a  general  relationship  between  ethnicity,  income,  group  grades, 
group  standardized  test  scores  and  total  numbers  of  students.  The  documentation 
responds  to  the  intent  of  the  Health  Manpower  Act  (the  authorizing  legislation  for 
HCOP  is  Section  740  of  the  Act,  as  amended  by  the  Health  Education  Extension 
Amendments  of  1992)  which  requires  that  HCOP  dollars  be  used  to  increase  the 
number  of  disadvantaged  students  entering  and  graduating  from  health  professions 
schools  (see  Appendix  5). 

The  aggregate  data  format  is  also  useful  for  framing  an  hypothesis  and  reinforcing 
intuitive  conclusions.  For  example,  Charles  F.  Whitten  M.D.,  Associate  Dean  of 
Curricular  Affairs,  Wayne  State  University  School  of  Medicine,  maintains  that  the 
AAMC's  projected  goal  of  3000  by  2000  (increase  to  3000  by  the  year  2000,  the 
number  of  URM  matriculants  into  health  professions  schools)  is  not  likely  to  be 
achieved.  He  maintains  that  the  goal  can  better  approximated  if  more  post 
baccalaureate  programs  are  funded  and  implemented.  Whitten  has  effectively  used 
aggregate  data  to  illustrate  that  there  has  been  neither  an  increase  in  the  applicant 
pool  of  URM  students  between  1981  and  1990,  nor  an  improvement  in  the  applicant 
pool's  quality,  as  depicted  by  ethnic  group  GPA  and  MCAT  mean  scores  (see 
Appendix  2).  These  students  —  those  who  are  now  or  will  be  eligible  for  application 
to  medical  schools  by  the  year  2000  —  are  already  fairly  far  along  the  educational 
development  pipeline. 

Appendix  6  illustrates  how  the  use  of  aggregate  data  by  past  and  present  HCOP- 
funded  post-baccalaureate  programs  can  be  used  to  portray  the  strong  association 
between  participation  in  their  programs  and  the  participant's  subsequent  entry  into, 
and  graduation  from,  a  health  professions  school  (see  Appendix  6).  This  data 
supports  the  recommendation  for  more  post-baccalaureate  -programs  which  can 
strengthen  the  competitiveness  of  the  existing  applicant  pool.' 

Notably,  the  aggregate  data  format  is  not  as  effective  a  mechanism  to  depict  the 
correlation  between  HCOP-funded  interventions  and  individual  student  performance. 
Additionally,  it  does  not  shed  additional  light  on  the  specific  academic  profiles  of  the 
students  who  gained  acceptance  and  the  students  that  were  rejected  by  medical 
schools.  Aggregate  data  lacks  the  level  of  detail  needed  in  order  to  design  and 
implement  the  most  appropriate  intervention  which  can  help  mitigate  the  gap 
between  the  minority  and  majority  matriculant  pools.    For  example,  little  of  the 
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following  insights  can  be  deduced: 

•  How  many  of  the  accepted  students  had  cumulative  science  grade  point 
averages  of 

>  3.00,  and  /or  a  MCAT  physical  science  subtest 
score  <  6? 

•  Which  of  the  accepted  students  with  cumulative  science  grade  point 
averages  <  3.00  and /or  total  MCAT  score  <  22  attended  a  Historically 
Black  College  or  University? 

•  Which  of  the  accepted  students  participated  in  an  HCOP  academic 
enrichment  program? 

Even  though  quality  aggregate  data  is  constructed  from  student-specific  data  sources, 
there  may  not  be,  as  has  been  the  case  for  many  HCOP  funded  projects,  sufficient 
incentives  for  HCOP  projects  to  focus  on  analytical  efforts  on  student-specific  data. 


Cohort  Data 

Several  HCOP  funded  programs  have  been  able  to  generate  generally  enlightening 
information  about  their  respective  applicant  and  participant  pools  by  using  cohort 
data.  From  the  research,  "cohort  studies  are  the  design  of  choice  for  studying  the 
course  of  ...  risk  factors,  because  they  can  be  readily  designed  to  be  longitudinal  and 
follow  a  group  of  subjects  over  a  period  of  time."34 

Grantees  should  be  able  to  analyze  this  level  of  data  at  a  relatively  small  dollar  cost. 
The  Progress  Report  review  revealed  that  some  grantees  have  utilized  this  format  as 
the  next  level  of  analysis,  after  presentation  of  aggregate  data,  to  draw  correlations 
between  selective  variables,  or  a  group  of  variables,  and  outcomes. 

For  example,  the  data  highlighted  in  Appendix  7  strongly  indicates  that  if  variables 
such  as  GPA  and  MCAT  scores  are  similar  for  two  subgroups  (cohorts),  the  grantee 
may  explore  other  variables  that  may  correlate  to  academic  performance.  The 
'dependent'  variable  which  differentiates  the  two  cohorts  -  in  this  Appendix  is 
'previous  participation  in  a  HCOP  academic  enrichment  program'  —  which  may 
explain  why  Cohort  A  performed  better  in  medical  course  work  than  Cohort  B. 


34  Dawson-Saunders,  Beth  and  Robert  Trapp .  (1994)  .  Basic  and 
Clinical  Biostatistics .  Appleton  &  Lange :  Norwalk 
Connecticut . 
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Over  the  years  grantees  have  utilized  cohort  data  to  provide  DDA  with  greater 
insights  into  which  variables  are  most  likely  to  affect  program  outcomes.  To 
understand  the  degree  (or  the  level  of  weight  each  variable  should  be  given)  to  which 
a  variable  will  impact  each  outcome,  some  grantees  have  collected  and  analyzed  more 
student-specific  data  as  indicated  in  Appendix  8.  From  the  progress  reports  it 
appears  that  when  a  multiplicity  of  variables  correlate  to  the  assessment  goal  —  which 
is  often  the  case  with  student-specific  data  —  multivariate  and  regression  analyses  is 
invaluable  in  gauging  their  relative  importance. 


Student-Specific  Data 

As  mentioned  previously,  aggregate  data  plays  a  significant  role  in  highlighting 
general  trends  in  program  outcomes,  developing  general  student  profiles  and  in  the 
presentation  of  information  supporting  the  need  for  the  continuation  of  Federal 
supplementary  education  programs,  such  as  HCOP.  Cohort  data  differentiates  and 
helps  categorize  information  into  relatively  small  representative  data  sets  which  are 
useful  in  reporting  more  specific  program  outcomes  than  would  be  possible  with 
aggregate  data. 

However,  it  is  through  the  collection,  assessment  and  utilization  of  student-specific 
data  that  verifiable  trends  and  outcomes  can  be  measured  effectively.  The 
information  gained  from  this  level  of  data  is  valuable,  not  only  in  accessing  program 
impact  but,  in  the  planning  and  development  of  the  HCOP-funded  interventions. 
Student-specific  data  provides  opportunities  for  continued  fine  tuning  throughout  the 
HCOP  intervention  rather  than  only  between  new  program  cycles. 

Some  grantees  have  used  student-specific  data  to  customize  skills  development 
strategies  in  light  of  individualized  student  needs,  to  establish  program  specific 
selection  criteria  and  to  link  intervention  approaches  (and  components  of  these 
approaches)  to  student  participation  in  the  HCOP  projects.  Appendix  8  illustrates 
how  one  grantee  applied  regression  analysis  to  determine  the  relative  weight  of  the 
contribution  made  by  a  student's  lavel  of  cognitive  skills  to  his  or  her  performance 
in  first  and  second  year  medical  school  courses.  Appendix  9  examples  the  use  of 
student-specific  data  in  the  customization  of  counseling  .efforts  within  an 
appropriate  intervention. 


Issues  and  Barriers  Faced  by  HCOP  Projects  in  Collecting  Useful  Data: 

HCOP  grantees  have  generally  done  a  good  job  in  collecting  and  communicating  the 
"aggregate"  data  solicited  within  the  grant  application  grids.  However,  this  level  of  data 
does  not  allow  for  an  effective  assessment  of  the  benefits  gained  by  individuals  from 
various  HCOP  supported  strategies.  Moreover,  the  "aggregate"  nature  of  the  data 
reported  makes  tracking  of  HCOP  participants  beyond  their  immediate  participation  and 
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subsequently,  throughout  the  health  professions  education  system,  very  difficult. 

The  review  of  progress  reports  completed  for  this  analysis  found  that  the  majority  of 
grantees  collect  a  wide  variety  of  student-specific  data.  This  information  is  often  derived 
from  student  completed  applications.  However,  the  spectrum  of  characteristics  tracked 
and  variables  measured  differs  greatly  from  one  HCOP  grantee  to  another.  Therefore, 
comparisons  across  programs  and  over  time  are  extremely  difficult  to  analyze. 
Additionally,  when  student-specific  information  has  been  reported,  the  research  has 
shown  that  it  is  often  not  well  defined.  For  example,  grantees  frequently  do  not  indicate 
whether  GPA  listing  represents  a  cumulative  science  GPA  or  an  overall  GPA.  Very 
often,  in  the  case  of  reported  standardized  test  scores  (eg  MCAT)  and  GPAs,  no  notation 
is  included  indicating  whether  the  information  came  from  official  or  unofficial 
transcripts,  and,  whether  the  scores /GPAs  had  been  validated  or  student  self- reported. 

HCOP  Work  Group  (HWG)  members  were  of  the  opinion  that  the  most  significant 
barrier  to  the  compilation  and  subsequent  analysis  of  student-specific  and  cohort  data 
is  related  to  resource  limitations.  One  of  the  critical  factors  lacking  in  many  HCOP 
projects  is  a  sufficient  number  of  staff  --  especially  staff  trained  in  the  development  of 
project  evaluation  methodology  and  quantitative  data  analysis  —  to  design  and 
implement  assessments.  Overlaid  on  these  limitations  are  the  demands  of  day-to-day 
program  implementation,  complicated  confidentiality  issues,  inappropriate  computer 
hardware  and  software,  and  institutional  bureaucracies  which  limit  the  access  of 
program  managers  to  key  information  sources. 

These  findings,  together  with  more  extensive  discussions  during  the  HWG  meeting,  led 
to  the  recommendations  that;  a)  a  data  collection  template  be  centrally  developed  and 
provided  to  each  Grantee,  and,  b)  that  a  data  clearinghouse  be  established  to  provide  a 
conduit  for  the  exchange  of  information  and  technical  assistance. 
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The  Importance  of  Collecting  Comparable  Data  (across  programs): 

It  is  increasingly  important  to  demonstrate  a  clear  cause-and-effect  relationship  between 
HCOP-funded  programs  and  the  ability  of  minority /disadvantaged  students  to  perform 
well  in  science  courses  and  on  standardized  exams  such  as  the  Medical  College 
Admissions  Test  and  the  United  States  Medical  Licensure  Examination.  Unfortunately, 
such  relationships  are  not  clearly  evident  from  the  available  data.  The  apparent 
emphasis  on  resource  allocation  in  support  of  program  interventions  over  quantitative 
analysis,  may  be  a  contributing  factor  to  the  limited  ability  of  programs  to  collect 
statistically  validated  data.  Moreover,  it  would  be  difficult  to  compare  data  across 
programs  because  programs  frequently  collect  dissimilar  data  elements. 

The  progress  reports  and  HWG  deliberations  indicate  that  for  comparisons  to  be 
developed  which  measure  results  'across'  HCOP  projects,  project  grantees  must  be 
willing  to  participate  in  the  establishment  of  a  "minimum  data  set."  Additionally,  for 
the  data  collection  to  be  successful,  each  Grantee  must  be  willing  to  integrate  these  data 
elements  within  each  project's  tracking  effort.  A  "minimum  data  set"  such  as  the  one 
proposed  under  Chapter  IV's  recommendations  will  facilitate: 

•  the  design  of  interventions  based  upon  accurate  representations  of  target  population 
and  grantee /program  characteristics; 

•  the  translation  of  these  characteristics  into  variables  for  input  into  selection  indices 
which  result  in  the  identification  of  key  indicators  of  projected  student  performance; 

•  the  monitoring  and  assessment  of  program  proficiency  and  cost-effectiveness;  and, 

•  the  drawing  of  statistically  validated  correlations  between  HCOP  interventions  and 
increases  in  the  number  of  minority  /disadvantaged  students  who  are  accepted  into 
health  professions  schools  and  graduate  in  a  timely  manner. 

Thus,  the  "minimum  data  set"  and  the  effective  analysis  of  the  data  generated  will 
provide  insight  into  which  interventions  may  be  worth  replicating  in  other  locations. 
Additionally,  the  information  gained  from  this  level  of  data  wilt  translate  into  a  more 
accurate  representation  of  the  aggregate  impact  of  HCOP  efforts'  on  a  national  basis. 
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However,    the    collection    and    utilization    of    this    data    carnes    with    it    two    key 
considerations: 

•  Safeguards  must  be  established  to  protect  confidential  information  provided  by 
students.  For  example,  a  student  identifier,  such  as  a  social  security  number,  can  be 
used  to  link  student  data  elements. 

•  The  Collection  of  Reliable  and  Validated  data  presents  a  significant  challenge  for 
grantees  whose  staff  time  and  quantitative  analysis  capabilities  vary  from  program 
to  program. 


In  addition,  to  maintain  consistency  among  discussants  around  the  issues  related  to  data 
quality  and  the  security  of  privileged  information  the  collection  effort  must  ensure  that 
the  following  criteria  are  met: 

•  Safeguards  -  moral  and  legal  considerations  demand  that  in  the  compilation  and  use 
of  any  data,  especially  student-specific  data,  shields  are  needed  to  ensure 
confidentiality. 

•  Reliability  -  relates  to  the  consistency  of  statistical  outcomes  measured  under 
identical  conditions.  For  example:  MCAT  scores  are  considered  reliable  because  the 
exam  is  administered  under  standardized /identical  conditions,  and  the  reported 
scores  after  each  test  administration  are  consistent  with  established  norms. 

•  Validity  -  the  extent  to  which  data  utilized  in  analysis  and  reporting  presents  an 
accurate  representation  of  program  outcomes  and  student  characteristics.  Validity 
also  applies  to  an  instrument  of  measurement  in  that  it  measures  what  it  purports  to 
measure. 


Additionally,  a  common  interpretation  of  these  criteria  is  critical  for  ensuring  a  'level 
playing-field'  among  the  HCOP  projects  for  the  purpose  of  data  collection  and  analysis. 
This  commonality  of  thought,  together  with  the  understanding  and  resources  needed  to 
effectively  collect  and  process  the  most  appropriate  level  of  data  for  a  given  task,  are 
critical  for  the  continued  evolution  of  HCOP  efforts.  The  following  recommendations 
reflect  the  research,  program  evaluation  and  HCOP  Work  Group  (HWG)  findings  which 
point  to  the  key  areas  of  supplement  needed. 
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RECOMMENDATIONS  FOR  FUTURE  DATA  COLLECTION  EFFORTS 

The  following  recommendations  are  based  upon  the  research  findings  and  analysis: 

#1  Develop  a  "universal  minimum  data  set"  for  high  school,  college  and  health 
science  school  programs  respectively,  and,  encourage  all  funded  programs  to 
incorporate  this  "data  set"  within  their  respective  applications.  Encourage 
Grantees  to  exchange  the  results  of  their  data  collections,  perhaps  through  the 
utilization  of  a  national  data  clearinghouse. 

#2  Develop  and  provide  all  HCOP  grantees  with  a  Common  Data  Template  (as 
presented. in  Chapter  III)  on  a  3.5/5.25  floppy  disk  and  encourage  all  programs 
to  utilize  it.  This  template  will  serve  as  a  guide  for  the  collection,  maintenance 
and  communication  of  minimum  data  requirements. 

#3  Encourage  grantees  to  collect  both  the  cognitive  and  non-cognitive  data  needed 
to  calculate  their  respective  selection  indices;  and  analyze  that  data  for 
performance  outcomes. 

#4  To  the  best  of  individual  capabilities,  encourage  HCOP  grantees  to  track 
participant  beyond  participation  in  program  and  when  possible,  throughout  the 
student's  health  professions  education. 
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CHAPTER  V 

A  SAMPLING  OF  INTERVENTIONS  DESIGNED 
TO  IMPROVE  THE  COMPETITIVENESS  OF 
MINORITY/DISADVANTAGED  STUDENTS 

Several  sources  concur  that  HCOP  funded  programs  have  indeed  been  successful  in 
directly,  or  indirectly,  assisting  thousands  of  minority  /disadvantaged  students  to  enter, 
and  graduate  from,  health  professions  schools.35  M  However,  researchers'  ability  to 
independently  assess  specific  HCOP  approaches,  let  alone  to  draw  comparative 
conclusions,  is  hindered  by  the  limited  availability  of  statistically  validated  data 
necessary  for  quantitative  evaluation.  Unfortunately,  as  explained  in  Chapter  II, 
relatively  few  HCOP  programs  have  published  student-specific  or  cohort  specific 
information  that  links  participants'  academic  year  grades  and  standardized  test  scores 
(post  HCOP)  with  the  learning  approaches  they  received  through  HCOP. 

Chapter  V  contains  descriptive  summaries  of  six  programs.  Each  of  these  programs  has 
(in  somewhat  varying  degrees  and  formats)  produced  statistically  significant  data  which 
strongly  suggests  correlations  between  its  interventions  and  students'  post-program 
academic  performance.  Published  findings  and  program  data  document  the  fact  that  a 
high  percentage  of  participants  (students  with  modest  to  poor  grades  and  standardized 
test  scores)  of  selective  academic  enrichment  programs  have  gone  on  to: 


receive  passing  grades  in  highly  competitive  science  and  math  courses; 


.37  38 


36 


35  Thomas,  S.  (1994).  "After  Two  Decades,  HCOP  Continues  to 
Create  Opportunities  in  Health, "  Black  Issues  in  Higher 
Education.  18-19. 

Testoff ,  A.  and  Aronoff  R.  "The  Health  Careers  Opportunity 
Program:  One  Influence  on  Increasing  the  Number  of 
Minority  Students  in  Schools  of  Health  Professions." 
Public  Health  Reports,  Vol.  98,  No.  3.  May-June,  1983. 

Bridgham,  Robert  and  Stacia  Scarborough.  ."Effects 
of  Supplemental  Instruction  in  Selected  Medical 
School  Science  Courses".  Academic  Medicine. 
67  (10)  ,  pp  569-571. 
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Fullilove,  Robert  E.  and  Treisman,  Philip  U. 
(1990).  Mathematics  Achievement  Among  African 
American  Undergraduates  at  the  University  of 
California,  Berkeley:  An  Evaluation  of  the 
Mathematics  Workshop  Program.  Journal  of  Negro 
Education.  59(3)  pp  463-478. 
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•  achieve  passing  scores  on  standardized  tests;39  *°  and, 

•  graduate  from  health  professions  schools  on  schedule.41  42 

METHODOLOGY  FOR  SELECTION  OF  PROGRAM  SUMMARIES 

There  is  relatively  little  information  in  the  educational  literature  which  provides 
statistical  validation  that  the  specific  learning  approaches  used  by  HCOP  funded 
programs  have  been  effective  in  improving  specific  cognitive  skills.  However,  there  is 
statistically  significant  information  that  clearly  shows  the  truly  important  correlation 
between  participation  in  selective  academic  enrichment  programs,  inclusive  of  some 
funded  by  HCOP,  and  improvements  in  academic  year  science  course  grades  and 
standardized  test  scores. 

The  methodology  used  to  select  the  programs  summarized  in  this  chapter  emanated 
from  a  thorough  review  of  the  educational  literature,  HCOP  Grantee  Progress  Reports 
and  recommendations 

proposed  by  the  HCOP  Work  Group  (HWG). 

During  the  first  stage  of  this  analysis,  each  program  was  screened  for  information 
pertaining  to  intervention  methodologies,  and  instruments  used  to  measure 
improvements  in  grades  and /or  standardized  test  scores  that  could  be  ascribed  to  the 
program  as  a  whole  or  to  a  specific  sub-intervention. 


39  Blanc,  R.  and  Martin,  D.  (1984)  .  "An  Evaluation  of 
a  Preparation  Program  for  Medical  Students  Who 
Previously  Failed  Part  I  of  the  NBME  Examination" . 
Journal  of  Medical  Education.  Volume  59,  pp  667- 
669. 
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Frierson,  H.T.  (1984).  "Impact  of  an  Intervention  Program 
on  Minority  Medical  Students'  National  Board  Part  I 
Performance" .  Journal  of  National  Medical  Association. 
Volume  76,  pp  1185-1190. 
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(1994).    National    Center    for    Supplemental 
Instruction,  UMKC .  Kansas  City,  MO. 

Unpublished   Data,    Georgetown   University   School   of 

Medicine.  Washington,  DC. 
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Secondly,  programs  containing  the  elements  outlined  in  the  first  stage  were  meticulously 
screened  for  student  specific  and  /or  cohort  data  that  would  indicate  the  intervention 
methods  they  employed  were  effective  in  improving  regular  course  grades  or 
standardized  test  scores  of  minority  students  at  'academic  risk.' 

Thirdly,  the  data  and  conclusions  were  analyzed  to  determine  their  reliability  and 
validity. 

This  process  resulted  in  the  identification  of  a  group  of  key  characteristics  that  could  be 
used  to  identify  those  programs  exhibiting  statistically  significant  improvements  in 
participant  academic  year  science  and  math  course  grades  and /or  standardized  test 
scores.  Based  upon  the  premiss  established  in  Chapter  II,  statistically  significant 
increases  in  course  grades  and /or  standardized  test  scores  provide  indirect  evidence  of 
strengthened  cognitive  skills. 


The  following  characteristics  provide  a  framework  for  selection  of  the  programs 
summarized  in  this  Chapter: 

•  A  target  group  consisting  of  at  least  50%  under-represented  minority  (URM)  or 
minority /disadvantaged  students;  and, 

•  An  indication  that  the  intervention  could  be  readily  adapted  for  use  by  HCOP 
Grantees. 

•  The  evidence  of  program  impact  was  derived  from  control  group  comparative 
evaluations  (this  characteristic  was  not  universally  applied  among  the 
interventions  summarized). 


PROGRAM  SUMMARIES 

Academic  enrichment  programs  which  strive  to  assist  minority /disadvantaged  students, 
usually  fall  into  two  broad  categories:  - 

i 

•  structured  short-term  (3  to  8  weeks)  summer  programs;  and, 

•  structured  longer-term  (12  to  15  months)  programs  that  usually  operate 
during  the  academic  year  or  during  the  summer  and  academic  year. 
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For   the    Program    Summary    section   of   this    report,   each    of   the   selected    sample 
programs43  will  be  presented  in  the  format  of: 

A:  Program  Overview 
B:  Program  Outcomes 


Xavier  University,  SOAR  (Stress  On  Analytical  Reasoning)  Program: 

A.  Program  Overview: 

The  educational  literature  and  the  Xavier  University  HCOP  progress  reports  strongly 
suggest  that  the  HCOP  program44  at  Xavier,  the  Summer  Science  Academy,  has  been 
effective  in  cultivating  and  nurturing  critical  thinking,  problem-solving,  and  test-taking 
skills  of  minority /disadvantaged  students  pursuing  careers  in  the  health  professions. 
The  most  significant  evidence  of  a  program's  capability  to  cultivate  these  skills  is 
demonstrated  by  statistically  significant  increases  in  participant  grades  and  MCAT  scores 
which  aid  these  students  in  becoming  more  competitive  for  acceptance  directly  into  a 
regular  (full  curriculum)  freshman  medical,  dental  or  pharmacy  class.  The  Summer 
Science  Academy  is  open  to  minonty/disadvantaged  high  school  and  college  students 
who  express  an  interest  in  pursuing  a  health  professions  career  (Medicine,  Osteopathy, 
Dentistry,  Veterinarian  Medicine,  Optometry,  Podiatry  or  Allied  Health). 


43  More  extensive  details  for  selected  components  of  each 
program  can  be  found  in  Appendices  10-13. 

44  HCOP  contributes  to  the  funding  support  of  the  Xavier 
Summer  Academy.  HCOP  funding  is  earmarked  for  Project 
SOAR,  the  component  of  the  Summer  Academy  that  targets 
rising  high  school  seniors  and  entering  college  freshmen. 
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The  Summer  Science  Academy  is  a  multi-tiered  and  multi-year  summer  program  which 
assists  URM  students  at  various  stages  of  the  educational  pipeline. 

•  Math  Star,  course  work  focuses  on  providing  URM  students  with  a  basic 
understanding  of  the  principles  of  algebra  and  trigonometry. 

•  Biology  Star,  course  work  provides  URM  students  with  an  introduction  to 
vertebrate  and  invertebrate  science. 

•  Chem  Star,  through  a  combination  of  classroom  lectures  and  laboratory  exercises, 
URM  students  learn  the  basics  of  the  chemical  sciences. 

•  Project  SOAR  (Stress  On  Analytical  Reasoning)  is  actually  the  last  stage 
of  the  Summer  Science  Academy's  academic  enrichment  process  and 
provides  rising  high  school  senior  and  rising  college  freshmen  students 
with  skills  development  courses  that  are  linked  to  college  level  math, 
biology  and  chemistry. 

Project  Soar  was  designed  as  an  interdisciplinary  effort  between  Xavier  University's 
math,  biology  and  chemistry  faculties.  In  addition  to  traditional  classroom  assignments 
with  strong  emphasis  on  vocabulary  and  reading  skills,  the  program  includes  small 
group  based  problem-solving  exercises,  peer  discussions  and  competitions  which 
develop  independent  thought  and  enhance  independent  learning.  Successful  program 
participants  serve  as  role  models  and  group  leaders.  Parental  involvement  has  also 
proven  to  be  a  key  factor  in  the  success  of  all  the  programs  sponsored  by  the  Summer 
Science  Academy. 


B.  Program  Outcomes 

Xavier  provides  the  following  evidence  that  supports  the  claim  that  its  HCOP  Program 
has  been  effective  (also  see  Appendix"  10): 

•  The  number  of  students  from  the  Xavier  program  who  gained  entry  into  health 
professions  school  almost  tripled,  from  an  average  of  44  per  year  in  1976-1978  to 
an  average  of  131,  between  1990-1993. 

•  Xavier  currently  (1994)  places  more  African- Americans  into  pharmacy  school  and 
into  medical  school  than  any  other  academic  institution. 

•  In  the  SOAR  program,  Nelson-Denny  Reading  and  PSAT  post-participation  tests 
show  consistent  gains  over  pre-parricipation  tests  each  year  from  1985  through  1993. 

•  The  Pharmacy  Pre-matriculation  Program  has  produced  statistically  significant 
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data  that  shows  more  students  have  been  passing  courses  such  as  pharmacy 
calculation  and  medicinal  chemistry  since  the  program  was  established. 

The  summer  program  participation  has  been  associated  with  a  reduction  in  the 
drop-out  rate  from  the  pre-medical  curriculum  -  from  43.6%  in  1984,  to  15.1%  in 
1993. 


The  Supplemental  Instruction  Program: 

A.  Program  Overview: 

The  Supplemental  Instruction  Program™  (SI)  was  designed  by  Deanna  C.  Martin,  PhD., 
during  1972  and  1973.  The  stimulus  for  the  program's  development  can  be  traced  to  the 
academic  difficulty  experienced  by  minority  and  other  disadvantaged  medical,  dental 
and  pharmacy  students  attending  University  of  Missouri  at  Kansas  City  (UMKC).  Initial 
funding  for  the  SI  program  was  provided  by  the  Office  of  Health  Manpower 
Opportunity  (OHMO),  through  the  Special  Health  Careers  Opportunity  Grant  (SHCOG) 
program  (see  Appendix  5). 

Following  the  success  of  the  initial  SI  program,  which  was  open  only  to  UMKC's  health 
science  students,  the  program  was  extended  to  other  schools  throughout  the  institution. 
Currently,  over  500  educational  institutions,  nationally  and  internationally  have  made 
the  Supplemental  Instruction  Program,  or  some  adaptation  of  it,  an  optional  part  of  their 
curriculum. 

After  a  rigorous  review  process  in  1981,  the  SI  Program  became  one  of  the  few  post- 
secondary  programs  to  be  designated  by  the  U.S.  Department  of  Education  as  an 
Exemplar)'  Educational  Program.  It  is  the  only  education  program  that  has  been 
certified  by  the  Department  of  Education  as  an  Educational  Retention  Program.  The 
National  Diffusion  Network,  the  dissemination  agency  for  the  U.S.  Department  of 
Education,  has  provided  federal  funds  to  support  a  national  SI  data  bank  which 
compiles,  analyzes  and  disseminates  reports  to  the  hundreds  of  educational  institutions 
across  the  nation  that  participate  in  the  network. 

The  following  critical  components  are  strongly  recommended  by  Si's  creators: 

•  The  SI  intervention  should  focus  on  high-risk  courses  (those  courses  with  an 
unacceptably  high  rate  of  grades  of  D,  F,  and  Withdrawals)  rather  than  on  high- 
risk  students.  In  this  way  the  program  allows  the  student  to  avoid  the  stigma 
associated  with  participation  in  traditional  academic  assistance  programs. 

•  Student  participation  should  be  voluntary  with  rare  exception. 
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Faculty  that  are  both  supportive  and  involved  need  to  be  identified. 

Students  (usually  successful  upper  class  or  graduate  students),  not  faculty,  should 
be  used  to  facilitate  the  learning  group. 

Skills  development  exercises  should  be  integrated  with  course  content; 

For  each  subject  matter  needing  reinforcement,  three  or  more  hours  per  week  of 
SI  should  be  allotted. 

The  length  of  each  SI  program  should  be  directly  proportional  to  the  length  of  the 
course  (quarterly,  semester  or  academic  year). 

Training  and  supervision  of  the  facilitator  should  be  mandatory. 


B.  Program  Outcomes: 

The  following  findings  are  statistically  significant  and  cut  across  race,  gender,  age  and 
comparable  academic  credentials  (also  see  Appendix  11): 

•  Students  participating  in  SI,  within  the  institution-targeted  high  risk  courses,  have 
exhibited  higher  mean  final  course  grades  than  students  who  do  not  participate 
in  SI. 

•  Students  participating  in  SI  within  targeted  high  risk  courses  succeed  at  higher 
rates  (withdraw  at  lower  rates,  and  receive  a  lower  percentage  of  D  or  F  final 
course  grades)  than  those  who  do  not  participate  in  SI  programs. 

•  Students  participating  in  SI  programs  persist  at  the  institution  (re-enrolling  and 
graduating)  at  higher  rates  than  students  who  do  not  participate  in  SI  programs. 

In  general,  SI  participation  produced  a  statistically  significant  positive  change  in 
academic  year  course  achievement.  The  average  effect  was  conservatively  estimated  as 
a  third  (and  is  more  likely  about  a  half)  of  a  standard  deviation  in  the  course  test  score 
distribution.   The  results,  outlined  above,  have  been  replicated 


54 


hundreds  of  times  within  a  wide  variety  of  higher  educational  settings,  both  nationally 
and  internationally.45 

A  HCOP  funded  post-baccalaureate  program  earmarked  for  URM  students  has 
successfully  incorporated  Supplemental  Instruction  into  its  curriculum  in  1992.  This 
program,  the  Advance  Baccalaureate  Learning  Experience  (ABLE),  reported  that  the 
minority  students  enrolled  in  the  program  showed  the  same  gains  as  other  SI  students 
when  their  grades  were  compared  to  the  grades  of  other  minority  and  non-minority 
students  who  did  not  attend  SI  sessions.46  The  effectiveness  of  SI  has  made  it  easier 
for  the  Michigan  State  College  of  Human  Medicine  to  maintain  an  admission  policy  that 
brings  a  diverse  student  population  to  the  college  and  that  ultimately  adds  to  the 
diversity  of  the  medical  profession. 

Lastly,  the  program  has  been  found  to  be  cost  effective  to  implement.  The  average 
expenditure  per  student  for  the  traditional  Supplemental  Instruction  Program  is 
approximately  $100. 


USMLE  Preparation  at  UMKC 

A.  Program  Overview: 

The  University  of  Missouri  at  Kansas  City  (UMKC)  preparatory  programs  provide 
educational  enrichment.  These  derive  from  original  funding  from  the  Office  of  Health 
Manpower  Opportunity,  under  a  Special  Health  Careers  Opportunity  Grant  (SHCOG). 
Although  no  longer  funded  by  HCOP,  all  UMKC  programs  continue  to  assist  minority 
students. 

Board  Preparation  (21  Weeks,  January-Tune).  This  program  assists  students  who  have 
previously  failed  the  Step  1  of  the  United  States  Medical  Licensure  (USMLE) 
examination,  especially  those  with  repeated  failures.  Board  preparation  incorporates  the 
standard  Supplemental  Instruction  (SI)  Program  but  differs  in  key  respects: 


45 


46 


Review  of  Research  Concerning  the  Effectiveness  of  SI  from 
the  University  of  Missouri-Kansas  City  and  Other 
Institutions  from  Across  the  United  States  .  (1994)  . 
National  Center  for  Supplemental  Instruction,  UMKC.  Kansas 
City,  MO. 

Bridgham,  R.  and  Scarborough,  S.  (1992).  "Effects  of 
Supplemental  Instruction  in  Selected  Medical  School 
Science  Courses."  Academic  Medicine  Journal.  67(10),  569- 
71. 
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it  is  designed  to  review  courses  already  taken; 

it  is  designed  to^prepare  students  for  the  USMLE  and  not  for  a  course 
grade; 

participation  in  all  program  components  is  mandatory; 

interactive  learning  groups  are  generally  composed  of  no  more  than  10 
students; 

facilitator  led  interactive  groups  meet  an  average  of  60  hours  per  week  in 
contrast  to  the  3  hours  per  week  allocated  for  SI  groups; 

there  are  preview  sessions  prior  to  lectures  in  basic  science  medical  school 
courses; 

question  based  tutorials  which  focus  on  the  syntax  and  the  semantics  of 
both  board  type  questions  in  addition  to  the  scientific  content. 

Students  who  have  previously  failed  Step  1  are  most  receptive  to  the  UMKC  FirstPrep 
curriculum  design.  This  is  particularly  true  for  "at  risk"  students  since  most  are  insecure 
about  their  academic  status  and  tend  to  be  fearful  of  adopting  new  or  different 
approaches  to  learning. 

Board  Preparation  (7  Weeks,  May-Tune).  FirstPrep  is  a  video-based  program  designed 
to  assist  students  who  are  preparing  for  the  June  USMLE  Step  1.  This  program  is 
offered  on  the  UMKC  campus  as  well  as  campuses  outside  Kansas  City.  Currently, 
FirstPrep  is  offered  at  20  sites  in  the  United  States  and  abroad.  A  four  week  program 
for  10-15  students  would  cost  approximately  $700  -  $1000  per  student,  inclusive  of  all 
teaching  and  materials  costs  (the  cost  does  not  include  student  stipends). 

Board  Preparation  (7  Weeks,  August-September).  This  program,  a  shorter  version  of  the 
January-June  offering,  is  designed  for  any  student  who  desires  a  structured  review  for 
the  September  USMLE  Step  1.  Special  consideration  is  given  to  students  who  failed  the 
previous  June  examination  with  scores  of  165  or  higher  and  expect  to  re-test  in 
September. 


Step  2  and  Step  3  USMLE  video-based  reviews  are  currently  in  preparation  for  use  in 
Kansas  City  and  at  other  campus  sites.  It  is  expected  that  these  reviews  will  be  available 
in  1996. 
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B.  Program  Outcomes: 

The  UMKC  programs  appear  to  be  the  only  standardized  test  preparation  programs  that 
offer  hard  statistical  data  indicating  their  success  at  significantly  increasing  the 
standardized  test  scores  of  the  majority  of  high  risk  students  who  participate  in  the 
program  (also  see  Appendix  12): 

•  Nearly  100%  of  program  participants,  who  have  previously  taken  National  Board 
Part  I  or  USMLE  Step  1  have  improved  their  scores  after  program  participation. 
The  majority  of  score  gains  are  statistically  significant. 

•  Nearly  all  students  who  did  not  pass  USMLE  Step  1  following  their  participation 
in  FirstPrep  were  able  to  pass  on  the  next  administration  after  participating  in 
the  Board  Preparation  (7  Weeks,  August-September)  program. 

•  The  brief  FirstPrep  program  is  not  designed  to  fill  in  large  knowledge  gaps 
in  a  short  period,  nor  will  the  FirstPrep  program  likely  enable  someone 
with  very  low  scores  to  succeed  on  re-test. 


The  Georgetown  Experimental  Medical  Studies  (GEMS)  Program: 

A.  Program  Overview: 

The  Georgetown  Experimental  Medical  Studies  (GEMS)  Program  is  a  post  baccalaureate 
program  that  currently  only  operates  during  the  academic  year.  The  primary  academic 
support  vehicle  is  a  modified  version  of  the  "Supplemental  Instruction"  program. 
Although  not  currently  funded  by  HCOP,  HCOP  funds  were  instrumental  in  enabling 
the  development  of  a  curriculum  that  emphasized  collaborative  learning.  Since 
implementation  of  this  curriculum,  the  GEMS  program  has  been  more  effective  in 
assisting  minority/disadvantaged  students  to  achieve  strong  pass  grades  in  all  basic 
science  medical  school  courses,  and  a  90-95  percent  pass  rate  when  USMLE  Step  1  is 
taken  for  the  first  time. 

The  participants  of  the  GEMS  program  do  not  pay  a  tuition  or  a  fee.  They  take 
approximately  50  percent  of  the  regular  freshman  medical  curriculum  during  the 
academic  year.  Lectures  on  embryology,  biochemistry  and  physiology  are  supplemented 
by  three  hours  of  monitored  self-learning  and  three  hours  of  facilitator-led  small  group 
interactive  learning  everyday. 
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The  primary  costs  of  this  program  are  those  associated  with  a  part-time  facilitator 
($20,000 /year)  and  with  copying  and  supplies 

($3,000 /year).  Special  faculty  support-time,  used  primarily  to  clarify  concepts,  averages 
20  hours  per  academic  year. 

B.  Program  Outcomes: 

The  GEMS  program  has  been  in  operation  for  16  years  during  which  time  the  following 
has  occurred: 

•  Two  hundred  and  sixty  (260)  minority  /disadvantaged  students  have  completed 
the  program.  The  completion  rate  has  improved  since  collaborative  learning 
strategies  were  integrated  into  the  curricula. 

•  Two  hundred  and  twenty-six  (226)  of  the  260  students  or  90  percent  of  the  GEMS 
students  have  gained  acceptance  to  an  American  medical  school.  The  medical 
school  acceptance  rate  has  also  improved  since  adoption  of  collaborative  learning 
approaches. 

•  Two  hundred  and  four  (204)  of  the  226  (90%)  who  gained  acceptance  to  a  medical 
school  have  either  graduated  or  are  still  enrolled  as  medical  students. 

•  Close  to  90  percent  of  all  GEMS  students  pass  all  basic  science  courses  and  Step 
1  of  USMLE  on  their  first  attempt.  The  first  attempt  pass  rate  of  USMLE  Step  1 
has  also  improved  since  adoption  of  collaborative  learning  approaches. 

The  GEMS  program  has  produced  measurable  data  which  strongly  supports  claims  that 
selected  HCOP-supported  learning  strategies  are  effective  in  making  students,  'at  relative 
academic  risk'  or  with  'marginal'  academic  credentials,  competitive  in  medical  school 
courses  and  on  Step  1  of  USMLE. 

It  is  important  to  note  as  well  that  GEMS  underscores  why  the  definition  of  student 
'risk'  must  be  program,  or  institutionally,  defined.  It  is  extremely  unlikely  that  students 
with  the  academic  profiles  of  those  enrolled  in  the  GEMS  program  could  perform  at 
acceptable  levels  if  they  had  to  start  their  medical  school  experience  by  taking  a  full 
course  load.  It  is  likely  that  their  academic  performance  has  been  enhanced  by  the  fact 
that  they  spend  50  percent  of  their  time  during  the  first  two  years  of  medical  school  on 
activities  devoted  to  the  better  understanding  basic  science  concepts  in  concert  with 
problem-solving  and  critical  thinking  exercises. 

Additionally,  the  format  and  outcomes  of  this  program  do  not  differ  greatly  from  those 
of  the  Video-Based  Supplemental  Instruction  program.  That  program  was  able  to 
generate  impressive  academic  outcomes  by  reducing  the  formal  academic  course  load 
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and  requiring  the  students  to  participate  in  monitored  collaborative  learning  activities 
for  time  periods  that  equaled  regular  course  time. 


Video-based  Supplemental  Instruction  and  the  Mathematics  Workshop  Program 

The  final  two  program  summaries  included  in  this  report  were  not  initially  designed  or 
implemented  with  HCOP  funds.  However,  the  statistically  validated  successes  each  has 
demonstrated,  together  with  the  fact  that  they  target  'students  with  traditionally  low 
academic  standings'  and  'historically  difficult  courses,'  indicates  that  they  should  be 
considered  for  greater  application.  The  first  of  these  approaches  is  called  VSI,  which 
stands  for  Video-based  (taped  or  assisted)  Supplemental  Instruction.  VSI  has  been 
adapted  by  numerous  supplemental  education  programs  since  its  inception  in  an  effort 
to  assist  college  athletes  in  attaining  passing  grades. 

The  last  program  highlighted  is  the  Mathematics  Workshop  Program  or  MWP.  This 
approach  utilizes  small  study  groups  where  students  work  together  to  identify  responses 
to  challenging  mathematical  problem  sets. 

Both  of  these  programs  have  been  included  because  they  can  serve  as  guides  for  the 
integration  of  learning  strategies  which  would  result  in  an  enhancement  of  HCOP 
efforts. 


The  Video-based  Supplemental  Instruction  (VSI)  Program: 

A.  Program  Overview: 

Video-based  Supplemental  Instruction,  as  a  stand-alone  program,  evolved  from  the 
University  of  Missouri  at  Kansas  City's  (UMKC)  goals  of  improving  the  quality  of  the 
academic  enrichment  gained  by  its  varsity  athletes  (many  of  whom  were  minorities). 
The  results  presented  under  the  outcomes  section  come  from  Kansas  City's  initial 
implementation  of  the  program,  in  conjunction  with  a  course  on  Western  Civilization, 
in  the  Fall  of  1992.  The  key  components  of  the  VSI  approach -implemented  by  the 
University  of  Kansas  City  include  the: 

i 

•  Selection  of  a  well  respected  professor  who  teaches  a  historically  difficult  course 
(typically  20-30%  of  students  in  class  receive  grades  of  D  or  F); 

•  Videotaping  of  all  lectures  included  in  a  course; 

•  Assignment  of  additional  credit  to  the  class,  in  the  case  of  UMKC's  program,  six 
credit  hours  were  given  -  three  for  the  core  course  components  and  an  additional 
three  for  the  supplemental  instruction  included  in  the  overall  VSI  approach; 
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Administration  of  the  same  exams  to  both  regular  course  and  VSI  course  participants 
at  the  same  time  and  grade  the  results  against  the  same  standards;  and, 

Utilization  of  a  facilitator  who  has  extensive  familiarity  with  the  subject  matter  of  the 
course;  and,  who  is  experienced  in  the  techniques  of  VSI  and  collaborative  learning. 


B.  Program  Outcomes: 

During  the  pilot  phase  of  this  program,  18  students  enrolled  in  the  VSI  section  and  157 
students  were  enrolled  in  the  regular  course.  The  results  were  examined  with  respect 
to  several  variables  because  course  grades  alone  were  not  sufficient  to  measure  all 
aspects  of  the  program.   The  following  results  were  reported: 


Ninety-five  percent  of  the  VSI  group  earned  A  or  B  grades  and  none  received  D's  or 
failed.   In  comparison,  24  percent  of  the  regular  class  either  received  D's  or  failed. 

The  average  grade  for  VSI  students  was  3.6  (4  point  scale)  in  comparison  to  2.3  for 
those  students  in  the  regular  course. 

All  but  one  of  the  VSI  group  members,  including  students  on  academic  probation,  re- 
enrolled  in  the  university  for  the  following  semester.  In  contrast,  85  percent 
(including  only  45  percent  of  those  on  probation)  of  the  regular  class  re-enrolled  in 
the  university  for  the  next  semester. 


Mathematics  Workshop  Program  (MWP) 

A.  Program  Overview: 

The  literature  also  surfaced  information  about  an  evaluation,  conducted  between  1978 
and  1985,  of  The  Mathematics  Workshop  Program  (MWP)  at  the  University  of  California 
at  Berkeley  (UCB).  This  program,  which  was  a  component  of  the  University's 
Professional  Development  Program  (PDP),  has  particular  relevance  to  this  research 
because  of  its  focus  on  the  use  of  collaborative  learning  approaches  to  master  math 
concepts  pertinent  to  health  professions  curricula. 

The  MWP  was  billed  as  an  honors  program  that  recruited  first-year  students  of  all  races, 
though  typically,  80  percent  of  those  enrolled  were  African  American  and  Hispanic.  The 
designation  of  MWP  as  an  honors  program  was  an  important  part  of  the  strategy  in 
recruiting  and  motivating  students  to  participate  in  the  program.  This  designation 
avoided  the  stigma  of  participation  in  a  remedial  course,  encouraged  the  students  to  gain 
assistance  in  a  challenging  subject  area,  earlier  in  their  educational  process  and  assisted 
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them  in  building  self-esteem. 

Once  enrolled,  students  discovered  that  the  program  functioned  very  much  like  the 
discussion  section  that  accompanied  their  mathematics  lectures.  What  distinguished 
each  workshop  from  a  standard  mathematics  discussion  section,  was  the  fact  that  the 
workshop's  20-30  participants  were  organized  into  groups  of  5-7  students.  Each  group 
worked  together  for  approximately  two  hours,  twice  a  week,  on  work  sheets  containing 
carefully  constructed  and  challenging  problems. 

The  work  sheets  were  the  core  of  the  program's  efforts  to  provide  students  with  three 
benefits:  1)  the  skills  to  earn  a  final  grade  of  A  in  first-semester  calculus,  2)  a  foundation 
in  mathematics  that  will  enable  MWP  graduates  to  continue  to  excel  in  second  semester 
calculus  and  upper-division  mathematics  (without  the  assistance  of  the  program),  and, 
3)  opportunities  to  identify  areas  in  mathematics  knowledge  that  students  must 
strengthen  to  survive  and  excel. 


B.  Program  Outcomes: 

Prior  to  the  establishment  of  the  MWP,  Chinese  students  generally  excelled  in 
mathematics  while  a  large  percentage  of  the  African  American  students  failed.  In  1976, 
for  example,  40  percent  of  the  African  American  students  enrolled  in  Mathematics  1A 
earned  grades  of  D-plus  or  lower.  The  data  indicates  that  the  MWP  program  has  been 
associated  with  significantly  more  positive  outcomes  for  minority  students  (also  see 
Appendix  13): 

•  A  total  of  131  African  American  students  completed  the  course  in  the  five-year 
period  between  1973-77;  by  contrast,  the  succeeding  five  years  saw  a  dramatic 
increase  in  the  numbers  of  African  Americans  both  enrolled  in  and  earning 
passing  grades  in  this  course. 

•  The  failure  rate  among  students  prior  to  the  establishment  of  the  MWP  was  33 
percent.  The  failure  rate  after  the  establishment  of  the  MWP  ranged  from  3-5%. 
This  difference  was  significantly  significant  in  comparison  to  the  two  groups  of 
non-MWP  students. 


• 


MWP  students  were  two  to  three  times  more  likely  to  earn  B-Minus  or  better 
grades  than  non-MWP  students,  irrespective  of  the  year  enrolled. 


MAJOR  FINDINGS 

To  summarize,  the  research  presented  in  this  report  indicates  that  participation  in 
collaborative  learning  sessions  is  associated  with  earning  higher  course  grades,  greater 
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persistence  in  college,  higher  science  grade  point  averages,  and  higher  graduation  rates 
from  health  professions  schools. 

The  most  significant  interventions  are  those  that  can  be  shown  to  be  effective  in  assisting 
students,  who  are  at  relative  academic  risk,  to  obtain  grades  and  standardized  test  scores 
that  are  competitive  for  acceptance  into,  and  timely  graduation  from,  health  professions 
schools.  Nearly  all  of  the  approaches  demonstrating  validated,  statistically  significant 
evidence  of  impact  on  academic  year  grades  and  standardized  test  scores  included 
collaborative  learning  techniques  as  a  common  feature. 

In  regards  to  the  directive  under  Task  10  (evaluate  and  identify  specific  successful  HCOP 
program  approaches)  of  the  research  contract,  the  comparability  of  data  and  the  level  of 
tracking  is  not  sufficient  for  the  majority  of  HCOP  funded  programs  to  ascertain  which 
strategies  are  directly  responsible  for  improvements  in  participants  scores  and  grades. 


Educators  believe  that  collaborative  learning  approaches  enhance  a  student's  ability  to 
acquire  information  and  recall  it  at  some  later  time,  discover  and  solve  problems,  and 
use  reasoning  to  support  conclusions.  The  documented  effectiveness  of  collaborative 
learning  in  assisting  students,  who  are  at  relative  academic  risk,  to  obtain  competitive 
grades  and /or  standardized  test  scores,  coupled  with  the  low  per  student  costs 
associated  with  the  approach,  suggests  that  many,  if  not  most  of  the  HCOP-funded 
programs  should  consider  adopting  this  learning  strategy. 

Minority /disadvantaged  students  will  not  achieve  competitive  grades  in  math  and  science  until 
their  cognitive  skills  level  can  be  significantly  strengthened.  The  achievement  of  competitive 
grades  by  'relatively  at-risk'  students  participating  in  collaborative  learning  activities  suggests 
that  the  learning  approach  has  the  twin  ability  of  showing  students  how  to  acquire  and  use  the 
information  they  have  gained.  This  has  a  direct  correlation  to  the  interests  of  HCOP  projects 
in  targeting  their  resources  towards  those  intervention  strategies  which  are  most  likely  to 
positively  impact  the  skills  minority /disadvantaged  students  need  to  successfully  compete  for 
entrance  into  and  timely  graduation  from,  health  professions  schools. 
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CHAPTER  VI 

ALL-INCLUSIVE  LISTING  OF 
RECOMMENDATIONS,  RELATED  GUIDELINES  AND 

RATIONALES 


The  HCOP  Work  Group  (HWG)  brought  together  for  this  analysis  feels  that  the 
Program's  future  success  is  closely  aligned  with  the  Division  of  Disadvantaged 
Assistance's  (DDA)  ability  to  encourage  Grantees  to  pool  data  and  experiences  leading 
to  the  ability  to  better  assess  the  effectiveness  of  their  collective  programming.  One  of 
the  overall  goals  of  this  collaboration  is  to  bring  together  statistically  valid  data  which 
aids  the  assessment  of  relative  levels  of  academic  risk,  identifies  effective  interventions 
and  productively  communicates  the  results.  The  HWG  discussion  focused  on  how  to 
best  provide  leadership  for  future  program  initiatives,  and  to  ensure  that  these  initiatives 
benefit  from  the  knowledged  gained  from  efforts  made  in  the  HCOP  projects  that 
preceded  them.  The  following  recommendations,  guidelines  and  rationales  reflect  the 
sentiment  of  the  HWG  participants  and  many  minority  medical  educators  who  are 
familiar  with  this  project. 

While  each  section  of  the  report  provided  targeted  recommendations,  those  included  in 
the  Introduction  were  developed  within  the  context  of  'global'  and  'systemic'  change,  as 
such,  they  overlapped  some  of  the  more  focused  recommendations  included  in 
subsequent  chapters.  This  overlap  was  intentional  in  order  to  ensure  that  all  proposed 
enhancements  to  the  program  were  interconnected  and  supportive. 

In  order  to  facilitate  the  utility  of  this  report,  the  guidelines  have  been  organized  into 
categories.  The  following  groupings  reflect  the  type  of  effort  each  guideline  would 
support:  Process  Guidelines  pertain  to  activities  which  would  enhance  the  operation  of 
HCOP  projects;  Cognitive  Guidelines  relate  to  skills  assessment  and  development 
impacting  those  abilities  which  are  traditionally  measurable,  such.as  science  knowledge; 
Non-Cognitive  Guidelines  reflect  the  issue  that  in  conjunction,  with  cognitives,  non- 
cognitives,  such  as  a  student's  drive  to  succeed,  should  be  incorporated  into  all  HCOP 
efforts. 
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CHAPTER  I:  INTRODUCTION 

#  1  Encourage  all  HCOP  Grantees  to  incorporate  HCOP  in  their  program 
name/title,  in  a  manner  which  is  appropriate  within  the  context  of  their 
individual  projects.  There  is  an  urgent  need  for  Grantees  to  develop  practices 
that  better  enables  students,  advisors  and  resource  allocators  to  know  that 
HCOP  provides  funding  and  technical  support  for  the  program. 

Process  Guideline  #1-A: 
Grantees  should  minimally  integrate  HCOP  into  their  title,  for  example:  The  *** 
HCOP  project,  or,  The  ^_  project,  funded  or  partially-funded,  by  HCOP.  The 
appropriate  mode  of  incorporation  may  reflect  the  relative  contribution  of  HCOP 
resources  in  complement  to  additional  funding  sources.  Within  this  context, 
program  participants  should  also  be  clearly  identified  as  HCOP  participants.  One 
way  to  approach  this  action  is  through  the  identification  of  participants  as  "HCOP 
Fellows"  (concept  is  detailed  further  in  Chapter  IV). 

Process  Guideline  #1-B: 
Grantees  should  ensure  that  when  appropriate  and  cost-effective,  any  form  of 
product  or  communication  (purchased  with  HCOP  funds)  clearly  indicates  HCOP's 
association  with  the  program.    This  should  pertain  to  letter-heads,  brochures  and 
program  applications,  as  well  as,  project  reports  and  publications. 

Rationale:  This  is  needed  to  increase  name  recognition  of  HCOP  efforts,  and,  to  aid 
in  the  tracking  and  reporting  on  the  formal  role  HCOP  plays  in  preliminary 
education,  facilitation  of  entry  and  retention  of  minority /disadvantaged  students  in 
health  profession  schools.  Additionally,  the  incorporation  of  the  HCOP  name  in  any 
form  of  communication  is  in  keeping  with  federal  regulations  and  standards  for  the 
identification  of  government-funded  efforts. 


#  2    Encourage  HCOP  Grantees  to  utilize  both  cognitive  and  non-cognitive  variables  in 
the  selection  and  the  matching  of  programs  to  participants. 

Process  Guideline  #2-A: 
Grantees  should  select  students  based  on  a  program  specific  standardized  index 
(discussed  in  Chapter  III  —  samples  are  provided  in  the  appendix  section)  that 
incorporates  both  cognitive  and  non-cognitive  variables. 
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Process  Guideline  #2-B: 
These  variables  should  be  weighted,  preferably,  by  means  of  regression  formulas 
that  place  relative  values  on  each  variable  which  the  project  feels  significantly 
influences  the  academic  performance  of  HCOP  participants. 

Process  Guideline  #2-C: 
The  weights  assigned  to  cognitive  and  non-cognitive  variables  should  be  set  in 
terms  of  relative  risk  levels,  including  those  which  reflect  the  perspectives  of 
individual  teaching  faculties  and  program  directors. 

Rationale:  The  educational  literature  confirms  that  a  combination  of  non-cognitive 
variables  coupled  with  cognitive  variables,  such  as  grade  point  average  and 
standardized  test  scores,  is  the  most  reliable  way  to  predict  most 
minority /disadvantaged  students'  readiness  to  compete  effectively  in  health 
professions  schools.  A  Selection  Index  based  on  weighted  cognitive  and  non- 
cognitive  variables  will  enable  Grantees  to  better  define  the  relative  risk  levels  of 
students,  which  will  vary  from  program  to  program.  Program  resources, 
particularly  financial  and  faculty  support,  go  far  in  determining  the  student  risk 
levels  a  program  can  reasonably  support.  A  Selection  Index  based  on  weighted 
variables  will  allow  each  Grantee  to  uniquely  define  student  risk  levels  in 
quantitative  terms.  Additionally,  clearly  defined  risk  levels  will  enable  Grantees 
and  DDA  to  better  assess  the  impact  of  HCOP  funded  interventions  on  students 
at  academic  risk. 


#  3  DDA  should  encourage  the  development  of  a  "universal  minimum  data  set"  (as 
discussed  in  Chapter  IV)  applicable  to  most  college  and  health  professions  school 
HCOP  programs  which  includes  both  cognitive  and  non-cognitive  variables.  DDA 
should  additionally  encourage  programs  to  include  questions  in  all  participant 
applications  which  will  elicit  the  necessary  data. 

Process  Guideline  #3-A: 
DDA  should  provide  all  HCOP  Grantees  with  a  computer  disk  that  contains  a 
Minimum  Data  Set  'Template'  (also  referenced  in  Chapter  IV)  and  encourage  all 
Grantees  to  utilize  it  to  facilitate  their  data  collection  efforts.. 


65 


Process  Guideline  #3-B: 

For  example,  the  template  should  include,  but  not  be  limited  to,  some  of  the 
following  data  elements  — 

•  Demographic  Data:  name,  gender,  date  of  birth,  ethnicity,  citizenship 

•  High  School  Data:  standardized  test  scores  (eg  SAT,  PSAT),  class  rank, 
course  grades 

•  Undergraduate/Graduate/Post-Baccalaureate  Data:  science  grade  point 
average  by  class  year,  cumulative  grade  point  average  by  class  year, 
standardized  test  scores  (eg  GRE,  MCAT,  DAT),  course  grades, 
institutional  competitiveness 

•  Intervention  Data:  participation  in  academic  enrichment  programs  (summer 
and /or  academic  year)  and  post-baccalaureate  programs. 

Rationale:  This  computer  disk  containing  a  Minimum  Data  Set  Template,  could  serve 
as  a  guide  and  tool  for  the  collection,  maintenance  and  communication  of  a  minimum 
of  standardized  cognitive  and  non-cognitive  data  deemed  important  in  defining 
minority/disadvantaged  student  academic  and  non-academic  needs.  It  would  also 
be  useful  in  predicting  their  potential  for  benefitting  from  participation  in  a  HCOP 
funded  program. 


#  4  Encourage  HCOP  Grantees  to  become  familiar  with  collaborative  learning 
approaches,  to  use  them  in  a  way  that  can  best  be  supported  within  their  existing 
resources,  and,  are  most  appropriate  in  meeting  the  needs  of  their  participants. 

Process  Guideline  #4-A: 
At  the  annual  technical  assistance  conference,  DDA  could  make  copies  of  the 
bibliography  of  this  research  paper,  available  to  prospective  applicants  who  are 
interested  in  doing  further  research  on  the  models  and  approaches  relative  to 
minority/disadvantaged  support  programs. 

Rationale:  This  is  needed  in  light  of  the  research  and  analysis  presented  in  this  report 
which  indicates  that  collaborative  learning  is  one  of  the  few  learning  approaches  that 
has  been  shown  to  be  statistically  effective  in  improving  the  higher  educational 
(college  and  health  professions  school)  grades,  standardized  test  scores  and  retention 
rates  of  Underrepresented  Minority  (URM)  students. 
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#5:  Support  the  establishment  of  a  "National  Student  Academic  Enrichment  Match 
Service  Center  (NSAEMSC),"  within  the  private  sector,  which  will  attempt  to  match 
a  student's  aspirations  (medical  school,  dental  school  etc),  academic  profile,  existing 
financial  circumstance,  grade  level  and  geographic  location,  with  the  curriculum  and 
support  service  programs  or  courses  which  best  address  his/her  needs. 

Process  Guideline  #5-A: 
NSAEMSC  would  encourage  student  applicants  to  voluntarily  submit  key  'selection 
indicators'  (detailed  in  Chapter  EI  -  such  as  GPAs,  standardized  test  scores,  grades 
in  specific  science  and  math  courses,  etc.)  to  the  "Center"  as  a  basis  for  the  match 
assessment  process. 

Process  Guideline  #5-B: 
NSAEMSC  would  encourage  Grantees  to  submit  specifics  related  to  their  respective 
curricula  that  identify  format(s)  (e.g.  facilitated  small  groups,  lectures),    size  of 
classes,  subjects  (including  the  amount  of  time  earmarked  for  each  subject),  average 
GPA  together  with  the  standardized  tests  scores  of  past  participants. 

Process  Guideline  #5-C: 
NSAEMSC  would  charge  students  a  fee  to  help  support  the  Center's  activities  in 
addition  to  seeking  foundational  grants. 

Rationale:  Currently,  students  are  in  no  position  to  effect  such  matches  on  their  own 
and  available  information  on  HCOP  funded  programs  does  not  provide  counselors 
or  mentors  with  enough  intervention-specific  information  to  allow  them  to  effectively 
match  the  individual  needs  of  students  to  available  enrichment  resources.  There  is 
no  question  that  hundreds  of  minority /disadvantaged  students  are  desperate  for 
technical  assistance  and  advice.  Many  do  not  know  where  to  direct  their  scarce 
resources  and  look  to  HCOP  supported  minority  medical  educators  for  guidance. 
Students  often  report  obtaining  contradictory  advice  from  those  sources.  This  is  true 
because  the  HCOP  funded  minority  medical  educators  are  themselves  unaware  of  the 
specific  curricula  or  learning  approaches  offered  by  HCOP  funded  programs  other 
than  their  own.  Even  when  that  information  is  known,  they  are  not  likely  to  know 
specifics  about  the  academic  profiles  of  the  students  who  participated  in  other  HCOP 
projects  or  specifics  on  the  effectiveness  of  those  programs.  There  is  an  urgency  to 
better  inform  the  students  and  their  advisors,  inclusive  of  HCOP  funded  minority 
medical  educators. 

A  more  effective  and  efficient  method  would  be  to  establish  a  national  match  service 
that  would  serve  a  function  not  unlike  that  of  the  National  Intern  and  Resident  Match 
Service.  It  would  not  have  to  be  sophisticated  or  costly.  This  service  is  urgently 
needed  since  students  are  not  in  a  position  to  effect  such  matches  on  their  own,  and 
currently,  the  available  information  on  programs  does  not  allow  counselors  or 
mentors  to  effectively  respond  to  individual  needs  of  students. 
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#6.    Establish  a  Standing  Guideline  and  Diagnostic /Intervention  Review  Committee 
which  includes,  to  the  extent  possible: 
a)  HCOP  Grantee  staff  b)  recognized,  technical  experts  in  student  testing,  c)  known 
authorities  on  cognitive  learning  strategies  and  d)  program  evaluation  specialists. 

Process  Guideline  #6-A 
This  can  be  accomplished  through  the  configuration  of  the  peer  review  panels  to 
ensure  that  each  of  these  skills  is  effectively  represented. 

Process  Guideline  #6-B 

Develop  guidelines  that  will  assist  Grantees  in  selecting  the  types  of  developmental 
skills  personnel  (see  Chapter  II),  most  likely  to  be  effective  in  improving  the 
cognitive  skills  of  "at-risk"  minority/disadvantaged  students. 

Cognitive  Guideline  #6-C 
Develop  guidelines  for  reading,  critical  thinking  and  problem-solving  skill  programs 
that  detail  at  minimum; 

a)  the  number  of  hours  of  instruction  to  be  provided;  b)  the  teaching  format  (e.g. 
size  of  student  group); 

c)  strategies  for  the  effective  linking  of  developmental  efforts  with  science  content 
domains;  and,  d)  methods  for  evaluating  the  effectiveness  of  cognitive  skills 
developmental  approaches. 

Process  Guideline  #6-D 
To  the  best  of  its  ability,  DDA  should  enable  the  committee  to  review  at  least,  a 
random  number  oi  grant  applications  and  progress  reports  each  year. 


Rationale:  A  Standing  Guideline  and  Diagnostic /Intervention  Review  Committee 
would  have  the  expertise  to  provide  advice  on  standards  and  guidelines  used  for 
student  selection,  program  design,  tracking  and  evaluation.  The  HCOP  Grantee  Work 
Group  (HWG)  indicated  that  many  Grantees  are  interested  in  collecting  more 
extensive  validated  data  and  participating  in  collaborative  efforts  designed  to  better 
analyze  the  effectiveness  of  their  respective  programs.  Unfortunately,  only  a  few 
Grantees  have  the  financial  or  manpower  resources  to  do  so.  Additionally,  the 
Committee  could  provide  a  source  of  technical  assistance  consultants  which  the 
programs  and  their  institutions  could  employ  to  assist  in  the  design  and  maintenance 
of  their  efforts. 
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#7.  Encourage  the  establishment  of  a  private  sector  based  "Data  Exchange  Clearing 
House  (DECH)"  which  compiles,  analyzes  and  disseminates  reports  related  to  data 
on  program  interventions  and  outcomes.  This  information  will  better  enlighten 
HCOP  grantees  about  the  cognitive  and  non-cognitive  factors  that  contribute  most 
to  the  success  of  HCOP  funded  programs  in  aiding  minority /disadvantaged 
students  to  become  more  academically  competitive  for  health  profession  school 
entry,  retention  and  graduation. 

Process  Guideline  #7-A: 
DECH  would  encourage  student  applicants  to  voluntarily  submit  data  (GPAs, 
standardized  test  scores,  grades  in  specific  science  and  math  courses  etc.)  to  a 
central  clearing  house. 

Process  Guideline  #7-B: 
For  those  HCOP  programs  in  need  of  assistance  in  processing  data  and /or 
developing  selection  indices,  the  clearing  house  could  also  provide  this  function. 

Cognitive  Guideline  #7-C: 
DECH  would  encourage  all  Grantees  to  submit  specifics  related  to  their  respective 
curricula  that  identifies  format  (e.g.  facilitated  small  groups,  lectures)  and  size  of 
classes,  subjects  and  the  amount  of  time  earmarked  for  each  subject,  average  GPA 
and  standardized  tests  scores  of  past  participants. 

Process  Guideline  #7-D: 
Distribute  fully  the  results  of  AAMC's  MCAT  Validity  Study  to  HCOP  instructors, 
counselors  and  student  advisors  to  assist  in  the  communication  of  realistic  short 
term  program  goals. 

Rationale:  The  HCOP  Work  Group  was  unanimous  in  the  opinion  that  DDA  should 
encourage  the  establishment  of  a  process  that  will  better  assess  the  effectiveness  of 
grantee  interventions  or  learning  approaches.  The  medical  education  literature  shows 
a  paucity  of  reports  that  confirm  the  effectiveness  of  the  various  learning  approaches 
used  in  HCOP  funded  programs  in  improving  the  academic  competitiveness  of 
students  who  participate  in  these  programs.  Even  though  intuition  suggests  many  of 
the  approaches  may  be  effective,  there  is  a  need  to  confirm  these  beliefs  with  hard 
data.  Such  a  confirmation  will  necessitate  HCOP  funded  programs  working 
cooperatively  and  systematically  together  to  compile  and  subsequently  analyze  the 
cognitive  and  non-cognitive  variables  which  influence  academic  performance. 

AAMC  studies  together  with  the  educational  literature  indicates  that  the  mean 
increase  for  student  cohorts  was  not  sufficient  to  move  from  one  cohort  to  another. 
Obviously,  within  each  cohort  group,  the  AAMC  study  did  show  individual 
examples  of  significant  increases. 
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#8.  PDA  should  encourage  the  development  of  a  tiered  academic  enrichment  system 
that  provides  highly  motivated,  minority/disadvantaged  students  an  opportunity 
to  cultivate  their  cognitive  skills  under  guidance  over  time.  In  addition,  PDA 
should  support  the  establishment  of  at  least  one  tiered  program  specializing  in 
cognitive  skill  development  within  each  region  of  the  United  States. 

Process  Guideline  #8-A 
Establish  the  levels  of  the  tiered  academic  enrichment  system  as  follows:  Level  1 
should  focus  on  basic  math,  reading,  writing  and  test  taking  skills  linked  to  general 
biology  or  physical  science  content  domains.  The  curriculum  design  for  this 
approach  should  place  a  priority  on  skills  development;  Level  2  should  build  on 
the  components  of  Level  1  and  include  additional  emphasis  on  problem  solving  and 
critical  thinking  skills;  and,  Level  3  should  proceed  along  the  continuum  of 
development  from  Levels  1  and  2  as  well  as  include  structured  exercises  to  improve 
the  participants'  potential  to  score  well  on  standardized  tests. 

Process  Guideline  #8-B 
Give  preference  funding  to  those  HCOP  applicants  that  can  demonstrate  an  ability 
to  affect  a  tiered  academic  enrichment  system  within  the  geographic  region  where 
their  program  will  operate. 

Process  Guideline  #8-C 
Establish  as  a  priority  the  development  of  intensive,  residential  summer  programs 
(6  to  8  weeks)  which  concentrate  activities  around  the  nurturing  and  enhancement 
of  cognitive  and  non-cognitive  skills  in  conjunction  which  math  and  the  sciences. 
These  programs  should  incorporate  a  minimum  of  twelve  hours  of  facilitator-led 
small  group  learning  sessions  and  utilize,  when  ever  possible,  video-taped 
instruction. 

Process  Guideline  #8-D 
Encourage  community  based  organizations,  community  colleges  or  undergraduate 
schools  to  be  participants  in  vertical  system  enrichment  programs  (those  which 
target  rising  high  school  seniors  and  /or  college  freshmen,  sophomore  or  juniors). 
Additionally,  have  applicants  include  evidence  that  their  program  has  a  formal 
linkage  agreement  writh  a  health  professions  school. 

Process  Guideline  #8-E 
Such  programs  should  be  targeted  to  the  highly  motivated,  bright  freshman  and 
sophomore  students  who  have  Science  GPAs  ranging  from  2.0-3.0  and  SAT  scores 
ranging  from  800-1100.  The  students  should  understand  that  it  may  take  2-3  cycles 
of  "a  summer  and  an  academic  year"  of  cognitive  skill  building  before  they  can  be 
reasonably  assured  that  they  would  be  competitive  in  a  health  professions 
educational  setting. 
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Non-Cognitive  Guideline  #8-F 
Admission  to  the  'base'  programs  should  be  restricted  to  students  who  evidence  a 
realistic  appraisal  of  their  need  to  improve  basic  cognitive  skills  as  determined  by 
interview  or  essay  and  non-cognitive  diagnostic  tests. 

Rationale:  More  examples  of  formal  collaboration  are  needed  between  HCOP  efforts 
and  between  the  individual  programs  and  the  educational  community  in  which  they 
operate.  By  placing  greater  focus  on  the  creation  of  tiered  relationships,  HCOP  can 
ensure  that  skills  development  does  not  end  at  the  door  of  an  individual  project.  Just 
as  critical  to  the  success  of  HCOP  programs  is  the  selection  of  students  who  are 
realistic  about  the  developmental  status  of  their  cognitive  skills.  In  particular, 
students  who  can  realistically  gage  these  skills  relative  to  the  developmental  status 
of  those  who  will  compete  with  them  on  standardized  exams,  for  an  acceptance  to  an 
health  profession  schools  and  for  grades  once  inside  the  school.  Many 
minority /disadvantaged  students  are  willing  to  accept  the  fact  that  there  is  no  quick 
solution  to  their  problem  and  are  willing  to  commit  2-3  years  working  on  cognitive 
skill  building  and  science  content  mastery  if  it  seems  likely  that  they  will  be 
competitive  for  health  professions  school  entry  at  the  end  of  that  period. 

Additionally,  summer  residential  programs  are  ideal  for  implementing  interventions 
that  promise,  or  have  been  proven,  to  be  effective  in  nurturing  the  development  of 
problem-solving,  critical  thinking,  conceptual  skills  and  standardized  test-taking  skills 
of  'at-risk  students.  Residential  HCOP  academic  enrichment  programs  for  high  school 
and  college  students  that  are  six  to  eight  weeks  in  length,  that  operate  somewhat  like 
'boot  camps'  in  format,  appear  to  be  effective  in  detecting  weaknesses  and  strengths 
in  cognitive  and  non-cognitive  attributes  of  the  program  participants.  Several  such 
programs  have  produced  data  that  strongly  suggests  they  may  be  effective  in 
introducing  students  to  developmental  processes  that  are  critical  to  the  eventual 
strengthening  of  problem  solving,  critical  reasoning,  conceptual  skills  and 
standardized  test  taking  skills.  Two  programs  which  have  received  much  publicity 
are  the  Drew  Summer  MCAT  Program  and  the  SOAR  program  at  Xavier  University. 


CHAPTER  II:  EFFECTIVE  LEARNING  APPROACHES. 

#1  Encourage  grantees  to  implement  learning  approaches  that  have  been  shown  to  be 
statistically  effective  in  improving  student  comprehension  and  conceptual 
understanding  of  lectures  and  textbook  material. 

Process  Guideline  #1-A 
At  the  technical  assistance  conference,  provide  prospective  Grantees  with  a  copy  of 
the  bibliography  from  this  research. 
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Process  Guideline  #1-B 

Circulate,  in  hard  copy  or  via  internet  (possibly  in  conjunction  with  technical 
assistance  from  the  Data  Exchange  Clearing  House  proposed  in  the  Introduction), 
examples  of  effective  approaches. 

Rationale:  Since  1972,  DDA  has  funded  hundreds  of  projects  designed  to  assist 
minority/disadvantaged  students  interested  in  pursuing  careers  in  the  health 
professions.  Unfortunately,  the  majority  of  the  projects  have  not  produced  statistically 
significant  data  that  confirms  a  correlation  between  these  learning  approaches  and  the 
improved  college  and  health  science  school  grades  or  improved  standardized  test 
scores  of  minority/disadvantaged  students.  Chapter  V  of  this  report  presents 
program  summaries  of  several  learning  approaches,  some  not  funded  by  HCOP,  that 
have  presented  statistical  evidence  supporting  their  ability  to  improve  the  academic 
competitiveness  of  minority  students.  The  1994  Evaluation  of  HCOP  Summer 
Programs  indicated  that  fewer  than  16%  of  those  programs  emphasize  the  learning 
methods  that  have  been  proven  to  be  effective.  This  information,  coupled  with  the 
fact  that  some  of  the  approaches  were  developed  with  HCOP  funding  support  more 
than  20  years  ago,  strongly  suggests  that  there  is  a  need  for  more  aggressive  efforts 
of  disseminating  historically  proven,  statistically  valid  intervention  models. 
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#2  Encourage  all  grantees  to  become  familiar  with  collaborative  learning  approaches; 
and,  to  adopt  those  approaches  which  can  best  be  supported  by  their  existing 
resources  and  are  most   appropriate  for  the  students  targeted  by  their  program. 

Process  Guideline  #2-A 
As  part  of  a  HCOP  'pipe-line'  approach  (noted  under  the  recommendations  listed 
in  the  Introduction),  integrate  intensive  six  week  summer  programs  within  a 
'tiered'    structure   which   includes    interactive    'boot   camp-like'    intervention 
programs. 

Cognitive  Guideline  #2-B 
Utilize  some  form  of  video-based  interactive /assisted  learning  (such  as  the  VSI 
program)  and /or  Supplemental  Instruction  (SI)  as  a  programming  component 
whenever  existing  resources  support  its  inclusion. 

Cognitive  Guideline  #2-C: 
Whenever    possible,    HCOP    course    work,    particularly    skills    development 
components,  should  be  directly  linked  to  science  domain  course  work.  In  addition, 
facilitator-led  small  group  interactive  learning  should  be  emphasized  whenever 
possible. 

Non-Cognitive  Guideline  #2-D: 

Using  LASSI  and  the  Non-Cognitive  Questionnaire  (NCQ,  a  revised  version  is 
currently  being  developed)  to  define  student  personality  types  and  to  better  match 
students  for  interactive  group  discussions  and  study  sessions. 

Rationale:  Based  on  a  review  of  the  literature  together  with  analysis  of  HCOP 
progress  reports,  these  program  components  have  been  linked  with  improvements  in 
student  academic  achievement.  Chapter  V  highlights  some  of  the  approaches  used 
by  programs,  which  appear  to  support  the  relative  benefits  to  be  gained  by 
incorporating  these  components. 


#3  PDA  should  minimally  earmark  funds  to  support  at  least  one  Grantee  in  each 
region  that  has  as  its  objective  the  improvement  of  HCOP  Participants'  scores 
on  the  science  and  math  portions  of  standardized  tests;  and,  makes  a 
commitment  to  use  collaborative  learning  approaches  to  improve  basic 
arithmetic  operations. 

Cognitive  Guideline  #3-A: 
Grantees  that  make  a  commitment  to  concentrate  on  improving  performance  of 
minority  students  within  the  math  portions  of  standardized  tests  through  the  use 
collaborative  learning  approaches,  should  commit  at  least  12  hours /week  towards 
the  development  of  basic  arithmetic  skills.    These  skills  include,  but  are  not 

73 


limited  to:  addition,  subtraction,  multiplication,  simplifying  algebraic  equations, 
proportional  analysis  of  algebraic  equations,  percentage  increase  or  decrease 
problems,  graphic  analysis  of  functional,  exponentials,  slope,  or  rate  of  change 
problems. 

Cognitive  Guideline  #3-B: 

Grantees  that  make  a  commitment  to  concentrate  on  improving  the  performance 
of  minority  students  within  the  science  portions  of  standardized  tests,  through 
the  use  collaborative  learning  approaches,  should  commit  at  least  12  hours/week 
towards  deductive  and  higher  forms  of  reasoning  in  conjunction  with  organic  and 
inorganic  chemistry,  physics  and  biology. 

Process  Guideline  #3-C: 
Within    this    context,    grantee    instructors    should    link    basic    arithmetic    skills 
development  efforts  with  the  content  domains  of  chemistry  and  physics. 

Non-Cognitive  Guideline  #3-D: 

It  will  not  be  enough  to  "just"  devote  more  time  to  basic  math.  Programs  need 
to  incorporate  collaborative/interactive  approaches  (similar  to  those  detailed  in 
Chapters  II  and  V  of  this  report)  into  their  interventions  which  lead  to  the 
enhancement  of  students'  confidence  in  attempting  math,  chemistry  and  physics 
problems. 

Rationale:  The  Med-Mar  MCAT  Minority  Student  Registry  suggests  a  significant 
number  of  medical  school  applicants  who  score  4  or  less  on  physical  science  MCAT 
sub-test  are  URM  students.  Since  the  physical  science  portion  of  the  MCAT 
emphasizes  chemistry  and  physics,  the  summer  program  curriculum  of  many 
HCOP  funded  programs  tend  to  emphasize  those  two  subjects  rather  than  on  basic 
arithmetic  operations  and  deductive  reasoning.  The  students,  and  too  often  the 
HCOP  project  personnel,  attempt  to  address  the  problem  of  low  physical  science 
MCAT  sub-test  scores  by  means  of  a  rapid  review  of  math,  chemistry,  and  physics 
during  a  6  week  summer  program.  The  students  are  exposed  to  many  complex 
science  and  math  concepts  (and  too  little  thinking  process)  over  too  short  a  period 
of  time.  Not  surprisingly,  students  often  leave  summer  programs  with  conspicuous 
weaknesses  in  basic  arithmetic  operations.  As  a  result,  they  obtain  essentially  the 
same  low  physical  science  MCAT  sub-test  scores  when  they  repeat  the  exam. 

An  analysis  of  the  commercial  and  non-commercial  MCAT  diagnostic  tests  reveals 
that,  in  spite  of  the  students'  college-level  math  grades,  most 
minority/disadvantaged  students  are  weak  in  the  basic  arithmetic  operations 
necessary  to  solve  chemistry  and  science  problems  efficiently  and  correctly. 
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The  approaches  outlined  in  the  guidelines  above  are  more  likely  to  improve 
students'  physical  science  MCAT  sub-test  scores,  especially,  if  they  continue  to 
practice  the  problem-solving  exercises  in  a  collaborative  learning  format  (e.g.  pair 
problem-solving). 


#4    Grantees    should    be    encouraged    to    sponsor    reading    interventions    that 
emphasize  evaluation,  application,  and  the  incorporation  of  new  information. 

Process  Guideline  #4-A: 
Applications  incorporating  reading  interventions    should  outline  the  following 
considerations    within    the    backgrounds    of    the    instructors    and    selected 
interventions: 

•  Specific  types  of  reasoning  and  problem  solving  skills  measured  by  standardized 
tests; 

•  The  number  of  hours  of  instruction  to  be  provided; 

•  The  types  of  methods  to  be  used; 

•  The  types  of  pre-  and  post-tests  to  be  used;  and, 

•  How  the  results  of  the  pre-  and  post- tests  should  be  interpreted. 

Process  Guideline  #4-B: 
Grantees  should  instruct  all  HCOP  participants,  who  are  preparing  for  an 
upcoming  standardized  exam,  to  read  the  AAMC  MCAT  Student  Manual 
carefully;  and,  Grantees  should  set-aside  time  for  a  test  expert  to  discuss  "key" 
points  in  the  manual  with  the  student  participants. 

Rationale:  There  is  no  easy  way  for  the  administrative  staff  of  HCOP  programs  to 
reliably  evaluate  the  potential  effectiveness  of  'reading  specialists'  or  'reading 
instructors.'  Unfortunately,  many  of  the  available  instructors . tend  to  emphasize 
vocabulary,  reading  comprehension,  and  reading  speed  at  the  expense  of  the 
evaluation,  application,  and  incorporation  of  new  information  items. 

The  MCAT  Student  Manual  (AAMC)  points  out  that  over  70%  of  the  MCAT  verbal 
reasoning  score  is  based  on  skills  in  evaluation,  application,  and  incorporation  of 
new  information  test  items.  Minority /disadvantaged  students  have  historically 
shown  weak  scores  in  these  areas. 

Student  Manuals  are  sent  to  students  when  they  register  to  take  the  exam  and  they 
offer  the  student  insights  into  the  format  and  rationale  behind  exam  questions.  For 
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example,  the  section  on  the  verbal  reasoning  indicates  that  the  intent  is  to  test  skills 
in  four  areas:  Comprehension,  Evaluation,  Application,  and  Incorporation  of  New 
Information.  Participants  need  to  better  appreciate  that  reading  interventions 
which  focus  on  evaluation,  application,  and  the  incorporation  of  new  information 
items  are  more  likely  to  achieve  higher  verbal  reasoning  scores. 

Experience  indicates  that  even  after  students  at  'academic  risk'  read  the  MCAT 
Student  Manual  carefully,  they  will  benefit  further  if  someone  proficient  in  all  three 
skill  areas  interprets  the  significance  of  the  information  to  them. 


#5  Grantees  should  be  encouraged  to  sponsor  writing  interventions  that  emphasize 
addressing  the  challenges  that  have  been  incorporated  in  the  writing  section  of 
the  MCAT  (i.e.  to  define  the  given  statement  as  thoroughly  as  possible,  to 
explore  the  meaning  of  the  statement  by  giving  a  contradictory  example  and 
to  discuss  ways  to  resolve  the  conflict  between  the  initial  statement  and  the 
contradictory-  example). 

Process  Guideline  #5-A: 
DDA  should  minimally  earmark  funds  to  support  at  least  one  Grantee  in  each 
region  that  has  as  its  objective  -  improvement  in  the  writing  portion  of  the  MCAT. 
This  Grantee  should  additionally  make  a  commitment  to  use  collaborative  learning 
approaches  in  conjunction  with  the  basic  writing  skills  exercises. 

Cognitive  Guideline  #5-B: 

Grantees  should  hire  writing  instructors  who  are  skilled  in  helping  students  to 
strengthen  their  skills  in  the  following  areas:  development  of  a  central  idea, 
synthesizing  concepts  and  ideas,  presenting  their  ideas  in  a  cohesive  and  logical 
manner  through  accepted  practices  of  grammar. 

Cognitive  Guideline  #5-C: 
Writing  instructors  should  link  basic  writing  skills  development  efforts  with  science 
content  domains. 

Process  Guideline  #5-D: 

Grantees  should  institute  mechanisms  that  will  support  participants  continued 
writing  skills  development,  over  a  12  month  or  longer  period. 

Cognitive  Guideline  #5-E: 
A  possible  scenario  for  ensuring  a  HCOP  Fellow's  writing  skills  continue  to 
improve  beyond  their  participation  in  a  HCOP,  would  involve  an  individual 
functioning  as  a  facilitator  (once  the  HCOP  is  completed).  The  facilitator  would 
establish  a  regular  schedule  for  evaluating  the  student's  progress  and  providing 
feedback,  which  would  ideally  be  done  every  three  to  four  months. 
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Rationale:   AAMC  and  other  research  have  shown  students  need  guidance  in 
addressing  three  tasks  that  are  asked  for  in  the  MCAT  essay  question,  e.g.:  1) 
definition  of  the  given  statement,  2)  providing  contradictory  examples  and, 
3)  discussing  ways  to  resolve  the  conflict  between  the  initial  statement  and  the 
contradictory  example. 

The  AAMC  Study  indicates  that,  on  average,  Writing  Sample  scores  for  repeat 
test-takers  do  not  significantly  increase  within  the  time  of  2  consecutive 
administrations  of  the  test.  The  MCAT  Validity  Study  coupled  with  pre  and  post 
writing  scores  obtained  on  MCAT  Diagnostic  Tests  strongly  suggest  that  one 
short-term  (e.g.  HCOP  summer  programs)  intervention  programs  does  not  provide 
enough  time  to  significantly  increase  students'  Writing  Sample  scores  on  the 
MCAT.  Interventions  for  the  Writing  Sample  section  of  the  MCAT  should  begin  as 
early  in  the  pre-med  educational  pathway  as  possible  and  should  be  reinforced  for 
as  many  months  as  possible.  Writing  samples  can  be  assigned  and  corrected  by 
mail,  HCOP  Grantees  could  pay  a  writing  instructor  to  provide  follow-up.  The 
Grantee  could  also  include  a  detailed  prescription  for  follow-up  in  the  HCOP 
Fellow's  exit  report  (noted  in  recommendation  #7  and  detailed  in  the  Chapter)  that 
could  be  used  by  student  or  the  student's  counselor  to  obtain  appropriate  follow 
up  support. 
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#6  Grantees  should  be  encouraged  to  implement  activities  in  HCOP  programs  to 
cultivate  non-cognitive  skills,  such  as  realistic  self-appraisal  and  positive  self 
concept. 

Guideline  #6-A: 
Utilizing  aspects  of  Collaborative  Learning  detailed  in  Chapter  II,  programs  should 
be  encouraged  to  target  approaches  to  enhance  both  cognitive  and  non-cognitive 
skills. 

Guideline  #6-B: 

Utilizing  the  Clearing  House,  Technical  Assistance  Conference  (listed  in 
recommendations  under  the  Introduction)  or  other  mechanisms,  model  programs 
and  approaches  for  improving  non-cognitive  skills  development  should  be 
communicated  among  HCOP  Grantees. 

Rationale:  Outcalt,  Rutala,  et  al  (Academic  Medicine  69  (1994):  577-582)  have  shown 
that  although  minority  /disadvantaged  students  entered  medical  school  with 
educational  disadvantages  and  scored  lower  on  "pencil  and  paper  tests",  their  clinical 
performances  on  clerkships  was  equivalent  to  majority  students.  Roessler  et  al 
(Medical  Education  53  (1978):  678  -  680)  have  also  found  that  the  addition  of  a 
"personality  variable"  to  a  cognitive  criteria  greatly  enhanced  the  prediction  of  mean 
basic  science  grades  and  NBME  scores.  The  staff  of  HCOP  funded  programs  readily 
acknowledge  the  importance  of  non-cognitive  variables,  used  in  conjunction  with 
cognitive  variables,  in  predicting  academic  performance  of  minority/disadvantaged 
students. 


#7  Establish  a  process  bv  which  HCOP  program  participants  will  leave  the 
program  with  diagnostic  results,  specific  information  about  the  programs 
learning  approaches,  outcome  performance  and  recommendations  designed  to 
enhance  future  development  of  cognitive  skills. 

Process  Guideline  #7-A: 
Grantees  should  provide  each  HCOP  Fellow  with  an  "Exit  Report"  which,  at  a 
minimum,  includes  the  results  of  cognitive  and  non-cognitive  assessment  tests,  a 
statement  about  the  specific  interventions  the  student  was  exposed  to,  and,  provides 
recommendations  for  follow  up. 


Rationale:  The  education  literature  supports  the  hypothesis  that  significant 
improvement  in  cognitive  skills  requires  nurturing  over  time.  HCOP  programs  can 
diagnose  cognitive  and  non-cognitive  weaknesses;  and,  can  subsequently 
demonstrate  how  the  weaknesses  might  be  addressed  over  the  life  of  the  program. 
Subsequent  development  of  these  skills  will  ultimately  require  that  the  student 
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practice  the  principles,  strategies  and  techniques  over  time.  Obviously,  this  phase 
of  the  skills  development  process  can  be  greatly  enhanced  if  this  practice  is 
monitored  and /or  facilitated,  by  an  instructor  or  advisor.  The  HCOP  Exit  Report 
would  provide  the  type  of  information  needed  to  enable  the  provision  of  post 
HCOP  program  support. 


CHAPTER  III:  SELECTION  OF  THE  HCOP  PARTICIPANT 

#1  Encourage  HCOP  Grantees  to  utilize  both  cognitive  and  non-cognitive  variables  as 
a  basis  for  the  selection  of  HCOP  participants  and  the  matching  of  those  participants 
to  the  programs  which  are  most  likely  to  benefit  them. 

Process  Guideline  #1-A: 
Encourage  HCOP  Grantees  to  define  and  weight  cognitive  and  non-cognitive 
variables  in  measuring  the  "relative  academic  risk"  of  the  applicants  within  the 
context  of  their  respective  program  characteristics. 

Process  Guideline  #1-B: 

HCOP  Grantees  could  provide  written  justifications  for  their  student  selection 
strategy  (identify  and  define  why  they  choose  each  variable,  determined  its  weight 
and  ultimately,  utilized  the  resultant  indicators  to  select  program  participants). 

Process  Guideline  #1-C: 
With  the  support  of  a  technical  advisory  committee  comprised  of  educational 
specialists,  statisticians,  and  selected  others  from  HCOP  funded  programs,  the 
Division  of  Disadvantaged  Assistance  could  conceptualize,  field  test  and  validate, 
a  model  HCOP  selection  index.  This  index  would  provide  cognitive  and  non- 
cognitive  parameters  for  the  admission  of  minority  students  into  HCOP  funded 
programs. 

Rationale:  The  educational  literature,  particularly  current  research  by  Sedlacek  & 
Prieto  [Academic  Medicine  (1990)  65  (3):  161-166]  and  Carmen  Webb,  M.D.  [Assistant 
Professor  and  Researcher  for  the  Office  of  Medical  education,  Medical  College  of 
Pennsylvania  and  Hahnemann  University],  points  to  a  tendency  by  health  professions 
schools  to  over  emphasize  cognitive  variables  in  the  selection  Of  students.  However, 
the  research  being  done  by  Sedlacek,  Webb  and  others  indicates  that  non-cognitives, 
appropriately  weighted,  and  used  in  conjunction  with  cognitive  variables,  are  useful 
predictors  of  minority  /disadvantaged  students  relative  risk  of  not  succeeding  in  the 
health  professions  education  process. 
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#2  Grantees  should  be  encouraged  to  collect  the  "minimum  data  set"  (discussed  in 
Chapter  IV)  together  with  additional  information  from  student  applications, 
letters  of  recommendation,  by  telephone  or  personal  interview  and  when 
possible  by  means  of  pre-admission  cognitive  and  non-cognitive  assessment 
tests. 

Process  Guideline  #2-A: 
Develop  and  provide  all  HCOP  grantees  with  a  Common  Data  Template  (detailed 
in  Chapter  V)  which  would  provide  the  programs  with  a  guide  on  how  and  what, 
information  to  collect. 

Process  Guideline  #2-B: 

The  'Data  Clearing  House'  (included  in  the  recommendations  listed  under  the 
Introduction)  could  provide  technical  assistance  to  programs  in  the  analysis  and /or 
utilization  of  this  data  for  program  design,  student  selection /program  matching,  as 
well  as,  project  monitoring. 

Rationale:  Several  medical  schools  have  developed  a  computerized  selection 
processes  based  on  cognitive  variables  that  have  been  weighted  by  means  of 
regression  analysis.  Sedlacek  is  currently  in  the  process  of  building  a  broad 
spectrum  listing  of  non-cognitive  variables  (which  are  applicable  to  the  health 
professions  environment)  and  will  ultimately  test  their  validity.  Some  of  the 
variables  that  will  be  used  in  a  Grantees  Selection  Index  will  differ  from  those 
included  by  other  Grantees;  however,  a  number  of  variables  will  be  common  for 
all  Grantees. 

Meanwhile,  studies  performed  on  undergraduate  (non-health  professions)  students 
indicate  that  cognitive  variables  tend  to  poor  predictors  of  academic  performance  of 
most  URM  students.  That  same  literature  indicates  that  non-cognitive  attributes  such 
as  realistic  self-appraisal  and  positive  self-concept  coupled  with  cognitive  variables 
are  better  predictors  of  academic  performance  than  cognitive  criteria  (e.g.  grade  point 
average  and  standardized  test  scores)  taken  alone  [Sedlacek  &  Prieto,  Academic 
Medicine  (1990)  65  (3):  161-166]. 

Use  of  a  "minimum  data  set  template"  would  establish  a  large  bank  of  student 
cognitive  and  non-cognitive  data  that  is  consistent  across  programs.  Grantees 
would  still  be  left  the  option  of  adding  HCOP-funded  program  or  program-specific 
cognitive  and  non-cognitive  data,  beyond  what  is  included  on  the  template. 
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#3  While  Grantees  should  be  encouraged  to  use  non-cognitive  variables  in  conjunction 
with  cognitive  variables,  they  should  be  strongly  cautioned  on  how  best  to  apply 
this  information  within  the  emotionally  charged  context  of  the  student  selection 
process. 

Non-Cognitive  Guidelines  #3-A: 
Within  the  context  of  participant  selection  or  denial,  decision  making  which  is 
subjectively  based  can  be  problematic.     Without  a  generally  accepted  rule  of 
measure  for  non-cognitives,  Grantees  should  consider  focusing  on  those  variables 
that  can  be  objectively  monitored  —  eg.  attendance  and  punctuality. 

Rationale:  Until  a  non-cognitive  testing  mechanism  can  be  developed  and  universally 
accepted,  any  non-cognitive  variable  which  can  not  be  objectively  assessed  and 
quantitatively  measured,  may  be  subject  to  challenges. 


CHAPTER  IV:  ASSESSMENT  OF  HCOP  PROJECT  DATA 

#1  Develop  a  common  set  of  'confidentiality'  guidelines  and  encourage  all 
Grantees  to  collect,  analyze  and  disseminate  student  related  data  within  the 
boundaries  of  these  guidelines. 

Process  Guideline  #1-A: 
Obtain  background  material  for  the  confidentiality  guidelines  from  the  Office  of 
the  Registrar  of  any  academic  institution  and  from  organizations  such  as  the 
Association  of  American  Medical  Colleges. 

Rationale:  The  analysis  presented  in  this  report  has  underscored  the  importance 
of  collecting  student  level  data,  that  is  comparable  across  programs.  This  is  needed 
in  order  to  better  define  which  high  risk  students  can  best  benefit  from  HCOP 
funded  programs  and  to  identify  the  approaches  most  likely  to  affect  that 
improvement.  Legal  and  ethical  considerations  demand  the  students  sign  an 
'informed  consent  agreement'  and  that  the  data  be  kept  confidential.  The 
Secondary  School  Science  Minority  Achievement  Registry  (S3MAR)  serves  as  an 
excellent  example  of  the  process  recommended  in  this  report:  The  Registry  lists 
student  data  emanating  from  questionnaires  which  were  collected  by  the  various 
program  directors  and  forwarded  to  the  AAMC  for  inclusion  in  the  S3MAR.  The 
Directors  also  maintained  informed  consent  signed  by  each  student  and  his  or  her 
parent  or  guardian,  on  file. 
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#2  Encourage  Grantees  to  collect  cognitive  and  non-cognitive  data  utilizing  a  limited 
number  of  assessment  instruments  that  have  been  field  tested  and  shown  to 
produce  outcome  data  that  is  valid  for  minority/disadvantaged  students. 

Non-Cognitive  Guideline  #2-A: 
The  Division  of  Disadvantaged  Assistance  should  devise  a  plan  that  limits  the  use 
of  HCOP  funds  to  the  purchase  of  a  limited  number  of  field  tested  and  validated 
non-cognitive  diagnostic  tests.  A  provision  should  be  made  that  the  data  collected 
from  such  tests  will  be  correlated  with  cognitive  selection  criteria  and  course 
performance  indicators,  to  establish  levels  of  academic  risk  (low.,  moderate,  and 
high)  for  performance  of  minority  students  enrolled  in  health  professions  schools. 

Non-Cognitive  Guideline  #2-B: 

Grantees  that  conduct  personal  interviews  or  on-site  screening  of  applicants  prior 
to  acceptance  should  be  encouraged  to  require  this  select  group  of  applicants  to  take 
one  or  more  of  the  DDA  approved  non-cognitive  assessment  tests.  This  should 
assist  in  the  personalization  of  the  course  and  in  efforts  to  match  students  for 
collaborative  interactions. 

Non-Cognitive  Guideline  #2-C: 
Grantees  which  do  not  come  into  personal  contact  with  participants  until  after  they 
have  matriculated  into  their  programs  should  require  each  participant  to  take  one 
or  more  non-cognitive  assessment  tests  before  beginning  the  program. 

Cognitive  Guideline  #2-D: 

The  Division  of  Disadvantaged  Assistance  should  devise  a  plan  that  will  allow 
HCOP  grantees  to  purchase  a  limited  number  of  field  tested  and  validated  cognitive 
diagnostic  tests.  A  provision  should  be  made  that  the  data  collected  from  such  tests 
will  be  correlated  with  HCOP  defined  cognitive  selection  criteria  and  course 
performance  to  establish  levels  of  academic  risk  (low., moderate,  and  high)  for 
performance  of  minority  students  enrolled  in  health  professions  schools. 

Cognitive  Guideline  #2-E: 
The  source  of  'key'  data  variables  such  as  grade  point  averages,  MCAT  scores,  DAT 
scores,  etc.,  should  also  be  notated,  when  writing  proposals  .or  progress  reports. 

Cognitive  Guideline  #2-F: 
Grantees  should  report  the  MCAT  sub-test  scores  of  medical  school  applicants  or 
medical  students;  and,  indicate  whether  the  MCAT  sub-test  scores  are  AMCAS 
validated,  student  self-reported,  or  transcript  validated.  In  those  instances  where 
AMCAS  validation  is  not  made,  Grantees  should  use  the  Minority  Med-Mar 
Registry  to  check  student  reported  sub-test  scores. 


82 


Process  Guideline  #2-G: 
The  Data  Clearing  House  could  serve  as  mechanism  for  the  distribution  of  cognitive 
and  non-cognitive  assessment  tests  and  could  provide  technical  assistance  in  their 
implementation  and  the  utilization  of  the  data  collected. 

Rationale:  There  is  an  urgent  need  to  design  a  process  that  yields  a  "minimum 
common  data  set"  that  cuts  across  the  many  programs  that  are  funded  by  HCOP. 
Pooled  data  will  have  the  potential  to  finally  yield  statistically  valid  outcomes,  an 
objective  which  is  currently  difficult  for  any  one  HCOP  funded  program  to  effect 
alone  because  of  the  relatively  small  number  of  students  that  participate  in  each 
program.  Additionally,  data  tests  which  have  been  validated  are  more  likely  to 
yield  information  of  worth  to  the  individual  programs  for  application  within  their 
design,  applicant  selection  and  impact  monitoring  efforts. 


#3    Encourage  Grantees  to  reflect  the  components  of  a  "minimum  data  set"  in  the 
questions  they  include  on  their  participant  applications. 

Process  Guideline  #3-A: 
Develop  and  provide  all  HCOP  grantees  with  a  Common  Data  Template  (in  Word 
Perfect)  on  a  3.5/5.25  floppy  disk  and  encourage  all  programs  to  utilize  it.    This 
template  will  serve  as  a  guide  for  the  collection,  maintenance  and  communication 
of  minimum  data  requirements. 

Rationale:  Reviews  of  HCOP  Progress  Reports  and  the  efforts  of  the  HCOP  Work 
Group  to  provide  5  year  priori  and  one  year  current  applicant  data  indicate  that  one 
of  the  most  significant  sources  of  foundational  student  data,  is  the  participant 
application.  All  Grantees  currently  collect  this  data,  in  varying  degrees  and  extent, 
however,  few  surveyed  effectively  used  a  consistent,  broad  spectrum  of  this 
information  to  track  and  access  program  effectiveness. 


#  4  Establish  a  HCOP  Fellows  Program  that  identifies  the  type  of  HCOP  program; 
and  also  indicates  how  many  different  HCOP  programs  the  a  student  has 
participated  in.  The  encourage  Grantees  to  use  this  information  to  enhance 
their  ability  to  track  and  subsequently  report  on  the  academic  progress  of 
former  participants  as  they  progress  through  the  health  profession  schools. 

Process  Guideline  #4-A: 
A  consistent  process  can  be  instituted  where  Grantees  identify  participants  in 
HCOP  funded  programs,  of  4  weeks  or  longer,  as  "HCOP  Fellows." 
The  following  is  an  example  of  how  such  designation  could  be  made:    HCOP 
Fellow-Level  One  =  Rising  high  school  seniors  and  rising  college  freshmen. 
HCOP  Fellow-Level  Two  =  Rising  college  sophomore  students.   HCOP  Fellow- 
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Level  Three  =  Rising  college  junior  students.  HCOP  Fellow-Level  Four  =  Rising 
college  senior  students.  HCOP  Fellow-Level  Five  =  Post  graduate  summer 
program  participant.  HCOP  Fellow-Level  Six  =  Post  graduate  academic  year 
program. 

Process  Guideline  #4-B: 

Encourage  Grantees  to  provide  each  HCOP  Fellow  with  an  "Exit  Report"  that 
minimally  includes  a  description  of  the  cognitive  and  non-cognitive  assessment 
results,  the  outcome  performance  related  to  program  interventions  and  the 
recommendations  for  follow  up. 

Process  Guideline  #4-C: 
Encourage  HCOP  Fellows  to  sign  a  release  which  allows  HCOP  projects  to  follow 
the  Fellow's  scores  and  GPA  throughout  the  student's  health  profession  educational 
process. 

Process  Guideline  #4-D: 

Develop  a  commitment  form  and  have  it  signed  by  all  HCOP  Fellows  who  elect  to 
do  so.  This  form  should  request  that  HCOP  Fellows  report  their  educational 
progress  back  to  their  sponsoring  HCOP  project. 

Rationale:  A  "HCOP  Fellows  Program"  would  enhance  student  tracking  efforts, 
engender  a  greater  sense  of  pride  withm  student  participants  and  give  the  national 
HCOP  program  greater  visibility.  By  providing  each  HCOP  Fellow  with  a  written 
exit  record  that  describes  the  cognitive  and  non-cognitive  profile  of  the  HCOP  Fellow 
at  the  time  of  entry  into  the  program,  including  details  on  the  type  of  learning 
experiences  and  how  the  student  performed  m  each  experience,  a  solid  blueprint  for 
the  student's  continued  growth  is  created. 

Additionally,  the  formal  designation  of  participants  as  HCOP  Fellows,  together  with 
the  signing  of  commitment  statement  forms,  will  empower  the  student  participants 
as  full  partners  in  the  HCOP  program,  enhance  student  self-esteem  (e.g.  MARC 
Fellows),  arm  students  with  specific  insights  about  their  academic  strengths  and 
weaknesses  status  and  generate  an 
"esprit  de  corps"  which  will  enhance  recruitment  efforts. 

On  example  of  an  application  of  "HCOP  Fellows"  is  in  conjunction  with  six  week 
summer  residential  programs  like  the  Drew  Summer  MCAT  program  and  the 
SOAR  program  would  be  ideal  for  Level  One  and  Level  Two  HCOP  Fellows.  They 
have  been  effective  starters  of  the  formal  cultivation  of  conceptual,  problem  solving 
and  reasoning  skills.  Once  skills  development  has  been  initiated,  both  programs 
have  developed  procedures  by  which  HCOP  Fellows  would  be  able  to  continue  the 
skills  nurturing  process  within  the  context  of  their  college  experience. 
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#5  Encourage  Grantees  to  participate  in  a  national  program  designed  to  better 
track  the  academic  performance  and  the  ultimate  career  path  of  former  HCOP 
student  participants. 

Process  Guideline  #5-A: 
Grantees  should  use  the  program-specific  cognitive  data,  together  with  their  non- 
cognitive  data  (using  HCOP-approved  tools),  as  a  consistent  basis  for  monitoring 
and  assessing  program  effectiveness. 

Process  Guideline  #5-B: 
The  information  described  in  Guideline  #5-A  should  be  transmitted,  via  the 
"minimum  data  set  template,  (elaborated  on  in  Chapter  IV)"  to  a  central  processing 
point,  such  as  the  Data  Clearing  House  (as  identified  in  the  Introduction) 


Rationale:  This  is  needed  to  ensure  the  availability  of  quality  comparable  data  which 
can  be  used  to  accurately  assess  HCOP's  impact  both  locally  and  nationwide,  and,  to 
identify  truly  effective  models. 


#6  Grantees  should  be  encouraged  to  track  the  academic  performance  of  past 
participants  and  relate  academic  outcomes  to  the  non-cognitive  attributes  defined 
while  a  participant  in  the  program. 

Process  Guideline  #6-A: 
The  first  step  of  this  effort  is  to  heighten  HCOP  name  recognition  in  conjunction 
with  program  efforts.    By  noting  HCOP's  involvement  clearly  within  the  title  or 
funding  sources  (as  noted  in  the  global  recommendations  included  under  the 
Introduction)  this  tracking  will  be  facilitated. 

Process  Guideline  #6-B: 

By  providing  students  with  exit  reports  or  report  cards  (as  detailed  in  the 
Introduction  and  Chapter  II)  which  describe  the  skills  development  interventions  in 
which  they  participated  and  what  is  needed  to  continue  developing  these  skills,  a 
continuum  will  be  initiated  which  allows  for  improved  tracking  beyond  HCOP 
participation. 

Process  Guideline  #6-C: 
Through  enhancement  of  the  HCOP  image  by  designating  participants  as  "HCOP 
Fellows"  (detailed  in  Chapter  IV),  students  are  'empowered'  to  be  at  least  partially 
responsible  for  the  reporting  of  their  own  progress.  Coupled  with  the  elective 
signing  of  academic  performance  reporting  release  forms,  the  stream  of  student- 
specific  data  is  likely  to  be  greatly  enhanced. 
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Rationale:  The  research  indicated  that  longitudinal  tracking  of  students  who  have 
participated  in  HCOP  programs  is  very  limited.  The  community  based  programs  are 
most  limited  in  their  ability  to  track  the  academic  progress  of  former  participants 
because  few  have  formal  linkages  to  academic  institutions  to  which  their  former 
students  matriculate.  HCOP  programs  sponsored  by  academic  institutions  generally 
are  able  to  track  the  academic  progress  of  former  participants  if  the  participants 
matriculate  into  academic  programs  at  the  same  institution.  However,  these  programs 
also  have  difficulty  in  obtaining  data  if  the  former  participants  matriculant  into  other 
institutions.  Therefore,  the  quality  and  quantity  of  tracking  data  can  be  greatly 
enhanced  if  former  HCOP  participants  take  greater  responsibility  in  reporting  their 
academic  progress  back  to  the  sponsoring  HCOP  program.  This  process  can  be 
encouraged  by  empowering  the  participant  as  a  HCOP  Fellow. 
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Appendix  #01 
HCOP  WORK  GROUP  MEMBERSHIP 
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William  Wallace,  Ph.D. 
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Director  of  Admissions 
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Texas  A  &  M  University 
College  Station,  Texas 

Lawrence  Dietz,  Ph.D. 
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Acting  Director 
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Acting  Deputy  Director 
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Rockville,  Maryland 

Arthur  H.  Hoyte  M.D. 

Director  Office  of  Minority 
Student  Development  and 
Assistant  Professor,  OB/GYN 

Georgetown  University  School 
of  Medicine 

Washington,  DC 

Aftab  Hassan  PhD 
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Washington,  DC 

David  L.  Taylor 
Senior  Research  Fellow 
Georgetown  University  School 
of  Medicine 
Washington,  DC 

James  Barnes  PhD 
President  Human  Systems 
Management  Corporation 
Washington,  DC 
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HCOP  WORK  GROUP  (HWG) 
MEETING  SUMMARY 

Between  June  3  and  4  of  1993,  a  group  of  twelve  representatives  from  broad  spectrum 
university  programs  met  in  Washington,  D.C.  for  an  intense  two-day  working  session. 
Each  of  these  individuals  represented  research  and  programming  efforts  which  had  long 
standing  association  with  the  Health  Careers  Opportunity  Program  (HCOP). 

The  goal  of  this  meeting  was  to  provide  the  ongoing  HCOP  research  effort  with  technical 
input  in  the  following  areas: 

I.  The  development  and  testing  of  Selection  Criteria  for  determining 
characteristics  of  minority  students  at  educational  levels  from  high  school 
seniors  to  health  professional  school  enrolles  who  are  most  likely  to 
improve  their  potential  to  become  health  care  professionals  by  participating 
in  Federal  educational  support  programs  such  as  the  Health  Careers 
Opportunity  Program  (HCOP). 

II.  The  development  of  a  Methodology  for  Analyzing  the  Effectiveness  of 
Specific  HCOP  Project  Interventions  approaches  to  improving  the 
problem-solving,  reading  conceptualization,  communication,  and  test- 
taking  skills  of  the  students  that  they  serve. 

The  lively  two-day  work  group  session  produced  many  significant  contributions  to  the 
research  effort  and  induced  extensive  discussions  around  the  key  areas  which  would 
provide  a  foundation  for  the  report.  The  following  short  summaries  highlight  some 
insights  presented  at  the  meeting. 


I.  Selection  Criteria 

For  the  purpose  of  the  discussion,  the  HWG  participants  were  asked  to  focus  on  the 
following  phrase  of  Task  7  of  the  research  contract:  "improve  their  [minority /disadvantaged 
students]  potential  to  become  health  care  professionals  by  participating  in  Federal  support 
programs  such  as  the  Health  Careers  Opportunity  Program". 

The  HWG  pointed  out  that  health  professions  schools  tend  to  be  at  the  end  of  the 
educational  pathway,  while  most  HCOP  programs  fall  earlier  along  this  pathway.  It  was 
the  participants'  view  that  HCOP  programs  might  consider  placing  some  weight  on  non- 
cognitive  criteria  in  conjunction  with  traditional  cognitive  criteria  such  as  grade  point 
averages  or  standardized  test  scores. 
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For  example,  the  HWG  members  felt  that  medical  schools  placed  an  excessive  emphasis 
on  MCAT  scores  when  considering  the  applications  of  under-represented  minority 
(URM)  students.  They  expressed  the  view  that  medical  school  admissions  committees 
did  not  adequately  take  into  account  a  number  of  factors  that  may  influence  the 
successful  medical  school  course  and  USMLE  performance  by  URM  students  with 
modest  MCAT  scores.  Much  of  the  discussion  centered  around  the  individual  program 
experiences  of  the  Work  Group  members  who  provided  the  following  observations: 

•  many  URM  students  successfully  complete  HCOP-funded,  post 
baccalaureate  programs  with  "A"s  and  "B"s  in  all  courses  in  spite  of 
modest-to-low  MCAT  scores;  and, 

•  URM  post  baccalaureate  students,  who  had  received  modest-to-low 
MCAT  scores,  gain  acceptance  to  medical  school  and  usually  pass 
all  medical  basic  science  courses  and  Step  1  of  USMLE. 

Given  these  programmatic  observations,  the  Work  Group  emphasized  the  need  to 
communicate  to  HCOP  programs  the  necessity  of  integrating  non-cognitive  skills 
development  in  conjunction  with  cognitive  skills  development  of  the  students  that  they 
serve.  The  discussion  was  enhanced  by  the  interjection  of  supportive  findings  from  the 
research  on  non-cognitive  attributes  carried  out  by  Dr.  William  Sedlacek,  and  insights 
gained  from  the  Association  of  American  Medical  Colleges'  MCAT  Validity  studies  — 
these  and  other  findings  are  detailed  more  extensively  in  the  body  of  the  final  report. 

The  discussion  on  selection  criteria  continued  beyond  the  directive  of  Task  7.  The  HWG 
emphasized  that  HCOP  approaches  (both  non-cognitive  and  cognitive  skills 
development)  alone  will  not  be  enough  to  reach  the  goal  of  increased  numbers  of 
minority /disadvantaged  students  being  accepted  into,  and  successfully  graduating  from, 
health  professions  schools.  Unless  the  admissions  committees  of  health  professions 
schools  are  willing  to  additionally  consider  the  skills  development  (both  cognitive  and 
non-cognitive)  advanced  by  the  applicant's  participation  in  HCOP  and  HCOP-like 
programs,  significant  increases  in  the  number  of  URM  students  accepted  into  these 
programs  will  not  occur,  at  least  in  the  near  future. 


II.  Analytical  Methodologies: 

The  HCOP  Work  Group  (HWG)  was  unanimous  in  the  opinion  that  before  they  could 
approach  a  discussion  on  the  identification  of  effective  methodologies  for  measuring  the 
impact  of  a  HCOP  intervention,  there  needed  to  be  a  greater  commonality  of 
understanding  of  what  the  inputs  to  this  process  were.  This  lead  them  to  the  initial  step 
of  outlining  the  key  questions  which  must  be  answered  before 

appropriate  methodologies  could  be  agreed  upon.  Some  of  the  more  far-reaching  of 
these  questions  include: 
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•  How  does  one  measure  the  impact  of  any  one  intervention? 

•  Should  the  HCOP  program  itself  be  viewed  as  an  intervention? 

•  Considering  that  many  HCOP  programs  do  not  ask  students 
about  previous  participation  in  a  HCOP-funded  activity,  is  it 
possible  that  the  impact  of  subsequent  HCOP  interventions 
could  be  inflated? 

From  the  discussion  of  these  and  other  questions,  it  was  obvious  to  the  Work  Group 
members  that  they  would  need  the  quantitative  and  qualitative  data  to  answer  the 
questions  they  had  posed,  and  to  develop  appropriate  methodologies. 

The  HCOP  work  group  (HWG)  members  recommended  that  the  first,  and  perhaps  most 
important  step,  was  to  perform  a  preliminary  evaluation  by  gathering  data  from  the 
programs  represented  by  the  members  of  the  group.  It  was  agreed  that  the  data  would 
cover  the  last  five  years  of  each  program's  operations.  As  part  of  this  preliminary  effort, 
the  group  also  recommended  that  each  program  administer  one  or  more  of  the  following 
diagnostic  tests  at  the  start  of  the  1994  summer  program  cycle: 

•  the  Learning  and  Study  Skills  Inventory  (LASSI)  test 

•  Sedlacek's  Non-Cognitive  Questionnaire 

•  the  Achievement  Diagnostic  Assessment  Test 

These  diagnostic  tests  were  selected  because  they  had  been  field  tested  and  validated  as 
being  appropriate  testing  instruments  for  minority/disadvantaged  student  populations. 


HWG  Concluding  Statements 

"The  Work  Group  believes  that  the  length  of  the  meeting  was  insufficient  to  adequately 
address  the  important  issues  that  surfaced  during  discussions  related  to  the  assigned 
tasks.  The  group  recommends  that  members  of  this  Work  Group  be  reconvened  in  the 
near  future  to  continue  this  process.  It  also  recommends  that  this  Work  Group  be 
expanded  to  represent  a  broader  cross-section  of  HCOP  program  types.  It  further 
recommends,  as  a  result  of  its  deliberation,  that  HCOP  view  this  charge  as  a  top  priority 
of  the  agency,  and  as  an  ongoing  process.  In  concert  with  this  view,  the  group  members 
will  commit  themselves  to  further  assist  in  developing  appropriate  guidelines  as  defined 
in  Task  7.  The  Work  Group  members  will  also  commit  themselves  to  assist  in  a 
preliminary  evaluation  exercise.  Each  member  of  the  group  with  readily  retrievable  data 
will  compile  that  data  and  forward  it  to  the  contract  staff  for  preliminary  evaluation. 
It  is  agreed  that  the  data  should  cover  the  last  five  years  of  each  program." 
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Appendix  #02 

Profile  of  the  Minority  Applicant  Pool 

for  American  Medical  Schools1 


National  data  indicate  that  a  significant  portion  of  educationally  and  economically 
disadvantaged  students,  many  of  whom  happen  to  be  minorities,  continue  to  test  lower 
on  average  than  do  most  majority  students.  For  example,  Whitten's  research  shows  that 
mean  MCAT  scores  of  African  American  medical  school  applicants  decreased  overall  for 
the  period  1978-1990,  and  were  less  competitive  than  those  of  the  majority  students. 

The  following  highlights  from  Whitten's  research  illustrates  the  concern  over  the 
quantity  and  quality  of  African  American  applicants  to  medical  school: 

Between  1978  and  1990,  there  was  an  overall  decrease  in  the  number  of  African 
American  applicants.  During  that  period,  there  was  also  a  cumulative  decrease  in  the 
grade  point  average,  the  biology  MCAT  score,  and  the  chemistry  MCAT  score  of  the 
African  American  applicant  pool. 


AFRICAN  AMERICAN  APPLICANTS 

<2.76  GPA 

BIOL  1-8 

MCAT 
CHEM  1-8 

QUANT.  1-8 

NO. 

% 

% 

% 

% 

1978 

2,564 

53 

88 

88 

89 

1981 

2,644 

53  . 

82 

85 

91 

1985 

2,428 

57 

76 

84 

92 

1989 

2,200 

43 

75 

83 

89 

1990 

2,355 

42 

75 

83 

91 

Whitten,  Charles  F.  (1991)  .  Proceedings  of  the 
First  National  Postbaccalaureate  Prog-rams 
Conference. 
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Equally  important,  between  1981  and  1990,  there  was  not  a  significant  change  in  the  number  of  African 
American  matriculants.  During  this  same  period,  there  was  an  overall  decrease  in  the  biology  and 
chemistry  MCAT  scores  of  African  American  matriculants. 


AFRICAN  AMERICAN  MATRICULANTS 

<2.76  GPA 

BOIL.  1-8 

MCAT  CHEM  1-8 

QUANT.  1-8 

NO. 

0/ 

/o 

0/ 

/o 

0/ 

/o 

0/ 

/o 

1978 

938 

30 

80 

80 

88 

1981 

1,001 

29 

73 

75 

88 

1985 

923 

27 

62 

/  5 

88 

1990 

1,055 

29 

62 

75 
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Whitten's  research  (1978-1990)  concludes  that  the  pool  size  as  well  as  quality  of  African  American 
applicants  to  medical  school  is  not  improving. 

It  should  be  noted  that  a  more  recent  study,  the  1995  Three  Year  Progress  Report  on  Project  3000  by  2000, 
concludes  that  "...URM  matriculants  have  increased  by  37%  since  1990,  ...  [and  that  the  project  is]  nearly  on 
target  for  reaching. ..the  goal  of  3000  URM  matriculants  by  the  year  2000. "  The  Progress  Report  additional  states 
that  "it  is  likely  that  the  growth  in  minority  science  bachelor's  degree  recipients,  and  minority  medical  school 
applicants,  in  part,  results  from  the  increased  popularity  of  medicine  as  a  career  choice  of  students  from  all  racial 
and  ethnic  backgrounds."2 

While  the  positive  trends  in  the  number  of  URM  applicants  and  matriculants  is  very  encouraging,  the 
HWG  expressed  caution  that  the  popularity-  of  medicine  as  a  career  choice  may  relate  to  current 
employment  trends  ie.  market  reductions  in  white  collar  job  opportunities.  In  addition,  the  HWG 
expressed  concern  that  the  increase  in  minority  matriculants  may  be  adversely  affected  by  the  current 
legal  and  political  climate  regarding  affirmative  action  policies.  Lastly,  the  1992  AAMC  TRENDS  report 
shows  that  biology  and  reading  MCAT  sub-test  scores  increased  (from  7.37  to  7.71,  and,  6.63  to  6.81 
respectively)  for  African  American  matriculants;  however,  because  the  comparative  scores  for  majority 
matriculants  remained  at  9.79  and  8.43  respectively,  the  increases  may  not  be  sufficient  to  maintain  or 
improve  the  ratio  of  African  Americans  who  successfully  matriculate  into  and  graduate  from  medical 
schools.3  Given  this  background,  the  HWG  reaffirmed  the  strong  need  to  continue  Federal  Support 
Programs  such  as  HCOP  and  PostBaccalaureate  Programs. 


Progress  3000  by  2000,  Year  Three  progress 
Report  (1995) .  Association  of  American  Medical 
Colleges.  Washington,  DC. 

Trends  in  Medical  School  Applicants  &  Matriculants 
(1992)   Association   of   American   Medical   Colleges 
Washington,  DC,  pp  IV- VI I . 
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Appendix  #03 
Measuring  Gains  in  Cognitive  Skills  Development 


Assessing  the  effectiveness  of  a  given  HCOP  approach  in  improving  the  cognitive  skills 
of  minority /disadvantaged  students  is  a  complex  challenge.  The  complexity  of  this 
process  is  accentuated  by  the  fact  that,  taken  as  a  group,  HCOP  programs  employ  a 
broad  spectrum  of  educational  and  learning  approaches.  As  a  result,  in  order  that  an 
assessment  can  be  made  of  gains  in  any  particular  are,  for  example  —  problem  solving  - 
-  analysis  must  follow  an  intricate  process. 

This  process  has  been  effectively  diagrammed  in  Champagne's1  Taxonomies  of 
Cognitive  Skills,  Thinking  Processes,  and  Academic/Assessment  Exercises  (see  attached). 
As  the  diagram  demonstrates,  it  is  an  extremely  difficult  process  to  predict  the  impact 
of  short  term  interventions  on  long  term  cognitive  skills  development. 


Champagne,  Audrey.  (1992).  Cognitive  Research 
on  Thinking  in  Academic  Science  and 
Mathematics  in  (Halpern,  Diane  ed.)  Enhancing 
Thinking  Skills  in  the  Sciences  and 
Mathematics.  Hillsdale,  New  Jersey:  Lawrence 
Erlbaum  Associates. 
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Appendix  #04 

A  Selection  Index  Utilized  by  the  Admissions  Committee 

of  a  Nationally  Recognized  Medical  School 


Most  institutions  of  higher  education  use  some  type  of  "indexing"  in  the  selection  of  their 
respective  students.  However,  the  majority  of  institutions  do  not  have  "rigid"  indices 
based  on  statistically  valid  formulae.  It  is  often  the  case  that  admission  committees  of 
these  institutions  make  acceptance  and  rejection  decisions,  particularly  those  involving 
under  represented  minority  (URM)  students,  on  the  basis  of  "generally  understood 
cognitive  and  non-cognitive  cut-off  variables."  For  example,  it  is  frequently  the  case  that 
admissions  committees  reject  any  student  with  a  MCAT  sub-test  score  of  4  or  less, 
and /or  any  student  who  had  a  grade  point  average  of  less  than  2.30. 

This  appendix  provides  an  overview  of  the  factors  that  are  currently  used  by  a  nationally 
recognized  medical  school  to  calculate  a  Selection  Index  for  applicants  (both  majority 
and  minority  students)  to  its  College  of  Medicine.  It  should  be  noted  that  the  Selection 
Index  consists  of  a  Cognitive  Index  and  a  Non-Cognitive  'Diversity'  Index. 

It  is  also  important  to  point  out  that  use  of  the  selection  index  has: 

•  increased  faculty  confidence  in  the  decision-making  of  the 
admissions  committee;  which  in  turn,  buttressed  faculty 
support  for  continued  minority  student  admissions;  and, 

•  enabled  the  admissions  committee  to  select  URM  students 
with  modest  academic  profiles  who  were  at  moderate-to-low 
risk  of  not  completing  their  medical  education. 


COGNITIVE  INDEX  (CI) 

A  regression  equation  was  developed  by  the  institution  to  yield  a  numerical  score  based 
on  the  following  cognitive  data:  cumulative  grade  point  average,  college 
competitiveness,  and  MCAT  scores.  Statistically  weights  were  applied  to  one  or  more 
of  the  equation's  variables  as  warranted  by  the  institution.  The  medical  school  utilizes 
the  following  cognitive  index  equation: 

CI  =  14.77  +  (.72  x  WGPA)  +  (2.52  x  MCAT  average) 
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WGPA  =  cumulative  GPA  (CGPA)2  x  college  weight 

College  weights  range  from  7.70  to  10.56 

All  schools  with  weight  below  7.70  would  receive  7.70  weight 

All  schools  with  weight  above  7.70  would  receive  their  actual  weight 

MCAT  average  =  Average  of  3  MCAT  scores 


EXAMPLE 

Method  of  computing  the  Cognitive  Index: 

Applicant  has  a  premedical  college  grade  point  average  (GPA)  of  4.5  and 
is  attending  a  college  with  a  weight  (WT)  of  9.2. 

Applicant's  MCAT  average  is  10. 

CI  =  14.77  +  [.72  x  (4.5  x  9.2)]  +  2.52  x  10)  =  69.78 
GPA  x  WT  MCAT 

AVG 


All  GPA's  are  converted  to  the  College  of 
Medicine's  5  point  grading  scale,  where  5.0=A, 
4.0=B,  3.0=C,  2.0=D,  and  1.0=E  (failing). 
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NON-COGNITIVE  DIVERSITY  INDEX  (DI) 

Similarly,  the  university  developed  an  equation  for  the  Non-Cognitive  Diversity  Index. 
The  numerical  score  is  added  to  the  cognitive  index  score.  Factors  contributing  to  the 
equation  for  the  diversity  index  include  the  following: 

I.  Minority  Status 

Cuban 

Hispanic  of  Pan  American  Origin 

Mexican  American 

American  Indian /Alaskan  Native 

Black  American 

Mainland  Puerto  Rican 


II.  Disadvantaged  Status 

Examples  of  students  with  a  disadvantage  include,  but  are  not  limited  to  the  following: 


a)  Those  who  maintained  a  part-time  or  full-time  job  during  college  out  of  necessity 
to  support  family  or  otherwise. 

b)  Those  whose  parents  are  less  educated  (e.g.,  do  not  have  a  high  school  diploma 
or  have  less  than  a  high  school  education.) 

c)  Those  individuals  who  have  suffered  from  prolonged  illness,  injury  or  extreme 
stress  that  has  interfered  with  their  academic  performance. 

d)  Those  individuals  who  have  a  physical  or  mental  disability. 

e)  Those  individuals  who  have  a  background  in  a  rural  or  underserved  region  of  the 
state. 


III.        Special  Achievements 
a)  Academic  Category 

1.  Advanced  degrees  (e.g.,  Ph.D.,  etc.) 

2.  Noteworthy  publications  (maximum  of  2  points  -  one  for  publication  while 
an  undergraduate;  one  for  publications  at  the  discretion  of  the  Screener) 

3.  Rigorous  academic  program 

4.  Honors  programs  or  honors  courses  where  warranted 

5.  Other  noteworthy  achievements  -  1  year  +  participation 
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b)  Non-Academic  Category 

Diversity  index  points  may  be  awarded  for  the  following  activities  providing  the  length 
of  participation  totals  one  year  or  more  in  each  area:  long-standing  achievement, 
leadership  in  campus  or  community  activities,  volunteer  work,  volunteer  undergraduate 
research,  music,  theater  and  those  who  served  for  the  United  States  on  active  military 
duty. 

In  response  to  the  need  for  primary  care  physicians  and  to  recognize  other  medical 
needs  in  the  community,  discretionary  Diversity  Index  points  may  be  awarded  to  show 
preference  for  characteristics  not  defined  elsewhere  in  the  guidelines  which  would 
further  enhance  the  diversity  of  the  class.  The  Screeners  may  award  up  to  2 
discretionary  Diversity  Index  points  for  activities,  providing  the  applicant's  participation 
time  totalled  1  year  or  more.  The  award  of  such  discretionary  points  must  be 
documented,  justified  by  the  Screener  and  approved  by  a  two-thirds  majority  vote  of  the 
Screeners  Subcommittee. 


SCREENERS  PROCEDURES  FOR  ENTERING  CLASS 

Candidates  who  are  offered  a  place  in  the  class  must  present  credentials  at  least  equal 
to  the  Cognitive  Index  (CI)  +  Diversity  Index  (DI)  [Selection  Index  (SI)  =  CI  +  DI]  listed 
under  Screeners  Guidelines.  The  CI  +  DI  was  selected  on  the  basis  of  a  statistical 
analysis  of  the  credentials  earned  by  the  class  entering  in  the  previous  year's  admission 
cycle.  The  Admissions  Committee  will  reach  one  of  the  three  final  decisions  for  all 
candidates. 


ADMIT 

ALTERNATE 


DENY 


The  candidate  will  be  offered  a  place  in  the  class  entering. 

The  candidate  will  be  selected  for  alternate 
status  and  given  a  specific  alternate  ranking  to 
be  determined  at  the  conclusion  of  the  review 
process. 

The  candidate's  application  has  been  reviewed  and  is  not 
competitive  with  candidates  who  will  be  offered  a  place  in 
the  class. 
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SCREENERS  GUIDELINES 

Minimum  cumulative  GPA  for  consideration:  3.30 

Minimum  1991  MCAT  numerical  average  for  consideration:  5.33 

Minimum  old  MCAT  numerical  average  for  consideration:  5.83 

Minimum  old  MCAT  Writing  Sample  Score  for  consideration:  J 

Minimum  SI  for  consideration:  69 

Decision  Recommendations  by  Selection  Index  Score  [Combined  CI  and  DI]: 

RESIDENT 

Deny  50.00  -  62.99 

Alternate  63.00  -  69.49 

Admit  69.50  or  greater 


NON-RESIDENT 

Deny  53.00  -  72.99 

Alternate  73.00  -  76.99 

Admit  77.00  or  greater 
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Appendix  #05 

The  Health  Careers  Opportunity  Program  (HCOPVSpecial  Health  Careers 

Opportunity  Grant  Program  (SHCOG) 


Begun  in  1972  as  the  Special  Health  Career  Opportunity  Grant  (SHCOG)  Program,  a 
Federal  initiative  known  today  as  the  Health  Careers  Opportunity  Program,  (HCOP)  was 
authorized  by  Title  VII,  Section  787  of  the  Public  Health  Service  Act,  as  amended  by 
Public  Law  100-607.  The  current  authorizing  legislation  for  the  HCOP  is  Section  740  of 
the  Act,  as  amended  by  the  Health  Education  Extension  Amendments  of  1992.  The 
purpose  of  the  HCOP  is  to  address  the  underrepresentation  of  minority  and 
disadvantaged  individuals  in  the  health  and  allied  health  fields  —  Allopathic  and 
Osteopathic  Medicine,  Dentistry,  Optometry,  Pharmacy,  Public  Health,  Health 
Administration,  Clinical  Psychology,  Chiropractic,  Podiatric  and  Veterinary  Medicine, 
graduate  programs  in  psychology  and  a  variety  of  allied  health  professions.  Experience 
has  demonstrated  that,  upon  completion  of  their  study,  HCOP  graduates  are  most  likely 
to  provide  clinical  and  other  health  services  to  underserved  rural  and  urban  populations. 

There  are  five  legislative  purposes  for  the  Health  Careers  Opportunity  Program3: 

•  Recruitment  -  Activities  designed  to  identify,  recruit,  and 
select  individuals  from  disadvantaged  backgrounds  for 
education  in  the  health  or  allied  health  professions.  For 
example,  motivational  activities,  distribution  of  information, 
exposure  to  role  models,  and  counselling. 

•  Preliminary  Education  -  Preprofessional  education  designed 
to  expand  the  academic  ability  and  otherwise  prepare  student 
participants  from  disadvantaged  backgrounds  during  their 
preprofessional  training  that  they  may  subsequently  complete 
the  regular  course  of  education  in  a  health  professions  school 
or  school  of  allied  health.  Such  education  must  be  offered 
prior  to  entry  in  a  health  professions  or  allied  health 
professions  school  and  may  not  include  courses,  already 
taught  as  part  of  the  regular  course  of  education  to  a  degree. 


1994   HCOP   Technical   Assistance   Manual. 
Division    of    Disadvantaged    Assistance 
Rockville,  Maryland. 
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Facilitating  Entry  -  Activities  designed  to  enhance  the 
competitiveness  of  student  participants  from  disadvantaged 
backgrounds  for  admission  to  health  professions  schools  or 
schools  of  allied  health,  such  as  improving  performance  on 
admissions  tests,  counselling  concerning  the  application 
process,  and  assisting  admissions  committees  in  the 
evaluation  of  disadvantaged  applicants. 

Retention  -  Activities  designed  to  help  student  participants 
form  disadvantaged  backgrounds,  who  have  been  accepted 
to  or  are  enrolled  in  health  professions  schools  or  schools  of 
allied  health,  to  complete  their  education.  Such  activities 
may  include  tutorial  assistance,  counselling,  and  assistance  in 
adjusting  to  the  environment  of  the  school.  Activities  may 
not  include  courses  already  taught  as  part  of  the  schools 
curriculum. 

Financial  Aid  Information  Dissemination  -  Activities  involved 
in  the  distribution  of  information  to  student  participants  from 
disadvantaged  backgrounds  about  financial  aid  available  in 
the  health  professions  schools,  schools  of  allied  health,  or 
schools  and  entities  which  provide  training  necessary  to 
qualify  for  enrollment  in  health  professions  schools  or 
schools  of  allied  health. 
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Appendix  #06 
Aggregate  Data  Supporting  the  Success  of  HCOP-funded  PostBaccalaureate 

Programs 


The  postbaccalaureate  programs  highlighted  in  this  appendix  have  utilized  aggregate 
data  to  strongly  suggest  that  their  programs  were  directly  responsible  for  hundreds  of 
underrepresented  minority  (URM)  students  gaining  acceptance  to  and  subsequently 
graduating  from  medical  schools  over  the  past  2  decades.  This  data  is  very  useful  in 
providing  an  overview  of  the  successful  outcomes  of  the  program. 


Post-Baccalaureate  Program  at  Creighton  University 

-  1975-1989  (14  Years) 

-  275  Students  Completed  the  Program 

-  230  Students  Accepted  by  a  Medical  or  Dental  School 

-  Acceptance  Rate:  84% 

-  214  Students  Graduated  or  Are  Still  in  Medical  or  Dental 
School 

-  93%  or  230  Students 


Post-Baccalaureate  Program  at  Wayne  State  University 

-  1970-1990  (20  Years) 

-  153  Students  Completed  the  Program 

-  132  Students  Accepted  by  a  Medical  School 

-  Acceptance  Rate:  87% 

-  110  Students  Graduated  or  Are  Still  in  Medical  School 

-  83%  or  132  Students 


Post-Baccalaureate  Program  at  Georgetown  University 

-  1977-1993  (16  Years) 

-  260  Students  Complete  the  Program 

-  226  Students  Accepted  by  Medical  School 

-  Acceptance  Rate:  90% 

-  204  Students  Graduated  or  Are  Still  in  Medical  School 

-  90%  or  204  Students  of  226  accepted 


102 


Appendix  #07 
The  Utilization  of  Cohort  Data  to  Predict  Program  Outcomes 


Several  HCOP  funded  programs  have  been  able  to  generate  generally  enlightening 
information  about  their  respective  applicant  and  participant  pools  by  using  cohort  data. 


The  data  highlighted  in  this  appendix  strongly  suggests  that  if  preadmission  variables 
such  as  grade  point  average  and  standardized  test  scores  are  similar  for  two  subgroups 
(cohorts),  the  grantee  may  explore  other  preadmission  variables  that  may  correlate  to 
performance  in  an  academic  year  course. 

The  data  for  each  cohort  follow  on  pages  104  and  105  respectively. 

Cohort  #1  consists  of  students  who  participated  in  a  HCOP  or  HCOP-like  Program,  or 
students  who  had  taken  a  post-graduate  Biochemistry  course. 

Cohort  #2  consist  of  students  who  did  not  participate  in  any  structured  HCOP  or  HCOP- 
like  academic  enrichment  program  during  or  after  college. 

With  the  exception  of  the  Science  GPA,  Cohort  #1  has  a  lower  average  for  the  Total 
GPA,  MCAT  scores,  and  College  Rating  columns  than  Cohort  #2.  Thus,  one  might 
expect  Cohort  #2  to  perform  at  an  academic  level  greater  than  or  equal  to  Cohort  #1. 

The  variable  which  seems  to  differentiate  the  two  cohorts  is  —  'participation  in  a  HCOP 
academic  enrichment  program'  or  previous  intervention.  This  variable  may  explain  why 
Cohort  #1  performed  better  in  the  medical  physiology  course  work  than 
Cohort  #2. 
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Appendix  #08 

Regression  Analysis  on  Student  Specific  Data 

to  Predict  Medical  School  Course  Grades 

This  appendix  illustrates  how  one  grantee  applied  the  statistical  procedure  of  regression 
analysis  to  student  specific  data  to  predict  course  grades  in  medical  physiology, 
biochemistry  and  embryology. 

Hypothesis 

Given  the  cognitive  profiles  of  students  considered  to  have  "high  academic  risk",  does 
participation  in  a  HCOP  or  HCOP-like  enrichment  experience  positively  correlate  with 
competitive  performance  in  difficult  medical  school  courses? 

Description  of  Cohorts 

Cohort  No.  1: 

Students  who  either  participated  in  a  HCOP  or  HCOP-like  program  or  completed  a  post- 
graduate course  e.g.  biochemistry. 

Cohort  No.  2: 

Students  who  did  not  participate  in  any  structured  HCOP  or  HCOP-like  academic 
enrichment  program  during  college  or  after  college. 

Design 

Criterion  -  freshman  medical  physiology,  biochemistry,  and  embryology  final  course 
grades. 

Predictor  -  Number  of  weeks  of  HCOP  or  HCOP-like  academic  intervention  support. 

Rationale 

Correlation  coefficients  established  in  several  studies  indicate  relatively  strong 
associations  between  Science  UGPA,  MCAT  Subtest  Scores  (particularly  the  Physical  and 
Biological  Sciences)  and  medical  school  first-year  grades  in  physiology,  biochemistry,  and 
embryology.4  5  6  7 


McGuire,  F.L.  "The  New  MCAT  and  Medical 
Student  Performance" .  Journal  of  Medical 
Education.  Vol  55,  1980." 
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This  analysis  will  determine  if  participation  in  a  HCOP  or  HCOP-like  experience 
significantly  improves  the  correlation  coefficients  for  academic  performance  in  medical 
school  physiology,  biochemistry,  and  embryology. 


Description  of  Regression  Equations 

The  data  from  two  cohorts  of  students  were  analyzed  by  means  of  three  regression 
equations.  These  equations  were  constructed  using  multiple  regression  procedures  for 
science  content  domains. 

The  physiology  course  grade  was  designated  criterion  1.  The  biochemistry  course  grade 
was  designated  criterion  2.   The  embryology  course  grade  was  designated  criterion  3. 

Participation  in  a  HCOP  or  HCOP-like  Program  was  used  as  a  predictor  for  academic 
performance  in  criterions  1-3. 


Results 

The   "HCOP   Enrichment"   positively   predicted    the   Physiology,   Biochemistry,   and 
Embryology  course  grades  for  both  cohorts  of  students. 


Association  of  American  Medical  Colleges. 
(1994) .  MCAT  Validity  Study.  Washington,  DC. 

Anderson,  J.  and  Mitchell,  K.  (1986) . 
Preadmission  Data  and  Medical  Performance  for 
Black  Medical  Students,  from  paper  presented 
at  AAMC  conference . 

Hall,  Michael  and  Stocks,  Michael.  (1995). 
Relationship  between  Quantity  of  Undergraduate 
Science  Preparation  and  Preclinical 
Performance  in  Medical  School.  Academic 
Medicine.  Vol  20  (3). 
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Appendix  #11 

National  Program  Outcomes  for  the 

Supplemental  Instruction  (SI)  Program 

Academic  achievement  for  students  of  differing  previous  academic  achievement. 

Data  were  analyzed  to  determine  the  utilization  and  effectiveness  of  SI  services  for  students  of 
differing  previous  academic  achievement.  Previous  academic  achievement  was  defined  by  high  school 
(percentile)  rank  and  mean  composite  score  on  a  college  entrance  exam  (e.g.,  American  College  Testing 
service).  Students  were  divided  into  quartiles  on  the  basis  of  their  mean  composite  ACT  score  as 
compared  with  other  UMKC  students.  These  data  warrant  the  following  observations: 

•  Students  in  the  bottom  quartile  group  used  SI  services  at  nearly  the  same 
rate  as  did  students  in  the  top  quartile. 

•  Despite  quartile  ranking,  Si-participating  student  earned  significantly 
higher  grades  than  their  non-participating  counterparts. 

•  Si-participating  students  in  the  bottom  quartile  and  the  middle  two 
quartiles  reenrolled  at  the  institution  at  significantly  higher  rates  than  their 
non-participating  counterparts. 


While  the  SI  and  Non-SI  groups  of  the  top  quartile  reenrolled  at  93 
percent,  the  Top  Quartile  Si-participants  received  a  significantly  higher 
mean  final  course  grade. 


It  is  noteworthy  that  SI  services  appear  to  meet  the  needs  of  students  with  a  wide  range  of  previous 
levels  of  academic  achievement  within  the  same  group  setting,  thus  reducing  the  necessity  for  the 
institution  to  provide  additional  and  separate  tutorial  programs. 


Academic  achievement  of  African-American  students. 

Table  1  describes  the  academic  performance  of  all  110  African- American  students  enrolled  in  12 
UMKC  College  of  Arts  and  Science,  School  of  Pharmacy  and  School  of  Basic  Life  Science  courses  that 
had  SI  attached  during  the  1987  Fall  Semester.  The  data  suggests  that  African- American  students  using 
SI  when  compared  with  Non-SI  students  of  the  same  ethnicity  had  a  significantly  lower  percentage 
of  D  and  F  final  course  grades  and  course  withdrawals,  and  earned  a  significantly  higher  mean  final 
course  grade. 
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Table  1 
Effectiveness  of  SI  With  UMKC  African-American  Students:  Fall  1987  (N=110) 


Group  Composition 

Number/Percentage  of 
Students 

Percent  D,  F, 
or  W 

Mean  Final 
Course  Grade 

Si-Participant 

39,35.5% 

31%' 

2.2" 

Non-SI  Participant 

71,64.5% 

46%' 

1.8" 

Level  ot  significance  ot  dilterence:(J.L)b  using  chi  square  test. 
Level  of  significance  of  difference:0.01  using  independent  t-test. 


Levels  of  SI  participation  and  academic  achievement  across  ethnicities. 

Data  analyses  were  also  used  to  find  the  utilization  and  effectiveness  of  SI  services  for  students  of 
differing  ethnicities.  A  sample  of  13  institutions  was  selected  for  analysis.  The  institutions  were 
selected  since:  they  had  numerous  Si's  in  place;  had  sufficiently  rigorous  data  collection  procedures; 
had  transmitted  their  data  to  us  in  a  timely  fashion;  they  represented  a  cross  section  of  institutions 
(3  two-year  public,  4  four-year  private  and  6  four-year  public).  Of  the  2,410  Si-participants  in  the 
study,  2,111  were  Caucasian  and  299  were  non-Caucasian.  Data  permit  the  following  observations. 
Students  in  each  Non-Caucasian  ethnicity  used  SI  services  at  equal  or  higher  rates  than  Caucasian 
students.  Despite  quartile  ranking,  Non-Caucasian  Si-participating  students  earned  higher  grades  than 
their  non-participating  counterparts  (p<.01).  Non-Caucasian  Si-participating  students  received  a  lower 
percentage  of  D  and  F  final  course  grades  and  withdrawals  (p<.05)  than  their  non-participating 
counterparts. 

It  is  noteworthy  that  SI  services  appear  to  meet  the  needs  of  students  with  a  wide  range  of  previous 
levels  of  academic  achievement  and  ethnicities  within  the  college  course,  thus  reducing  the  necessity 
for  the  institution  to  provide  additional  and  separate  tutorial  and  academic  support  programs. 


Ill 


Table  2a 
Participation  in  SI  By  Differing  Ethnicities:  1987(N=13  Institutions;  2,410  Si-Participants) 


Caucasian 

African  American 

Hispanic 

Asian /Pacific 

Native  American 

2,111 

174 

55 

42 

28 

Table  2b 
Effectiveness   of   SI   With   Differing   Ethnicity   and    Levels    of   Previous   Academic 
Achievement: 

Spring  and  Fall  1987(N=13  Institutions,  299  Non-Caucasian  Students) 


Group  Composition 

Percent  D,F  &  W 

Mean  Final 
Course  Grade 

SI 

Non-SI 

SI 

Non-SI 

All  Minority 

36%* 

43%* 

2.02** 

1.55** 

Lowest  Quartile, 
Minority 

Not  collected 

Not  collected 

1.87" 

1.35" 

Highest  Quartile, 
Minority 

Not  collected 

Not  collected 

2.64** 

1.97** 

"Level  ot  Significance  o 
**Level  of  significance  ( 

t  difference:  U.U5  using  (_fu-sq 
)f  difference:0.01  using  indepe 

uare  test, 
ndent  t-test. 
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Appendix  #13 
Program  Outcomes  for  the  Mathematics  Workshop  Program  (MWP) 


Comparison  of  Achievement  by  Subgroups 

Table  1  provides  a  comparison  of  academic  achievement  among  various  subgroups  of 
African  American  students  enrolled  in  Mathematics  1  A.  Irrespective  of  which  Academic 
Category  (column  1  of  table)  is  considered,  the  following  trend  emerges:  when  MWP 
students  are  compared  to  non-MWP  students,  within  any  given  category,  MWP  students 
scored  significantly  higher  than  their  non-MWP  peers,  even  those  non-MWP  students 
from  presumably  stronger  backgrounds.  The  most  significant  demonstration  of  this 
trend  can  be  seen  when  the  groups  are  examined  in  terms  of  their  respective  SAT 
Mathematics  scores.  Among  both  MWP  and  non-MWP  students  the  probability  of 
earning  a  grade  of  B-minus  or  better,  has  a  statistically  proven  correlation  with  SAT-M 
scores;  however,  when  MWP  students  with  SAT-M  scores  in  the  lowest  triad  of  the  score 
distribution  (i.e.,  scores  between  200-460)  were  compared  with  non-MWP  students  with 
scores  in  the  highest  triad  (i.e.,  scores  between  560-800)  there  was  no  significant 
difference  in  the  proportion  of  students  in  each  group  who  earned  final  grades  of  B- 
minus  or  better.  Overall,  irrespective  of  the  time  period  examined  or  the  manner  in 
which  students  were  grouped,  MWP  students  in  any  given  enrollment  category  were 
two  to  three  times  more  likely  to  earn  grades  at  the  higher  score  level  than  non-MWP 
students. 
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Table  1 

Proportion  of  African  American  Students  Earning  Grades  of  B-Minus  or  Better  in  Mathematics  1A 

Among  Selected  Sub-Groups  of  Students:  1973-77,  1978-82,  and  1983-84 


Academic 
Categories 

Historical  Control 
Group(  1973-77) 

MWP  Parti-opants 
(1978-82) 

Non-MWP  Participants 
(1978-82) 

MWP  Pam-cipants 
(1983-84) 

Non-MWP 
Participants*  1983-84) 

AU  Air  Amers  Taking  Math  1A 

N=131  22% 

N=188  54% 

N  =  143  16% 

N=43  58% 

N=141  23% 

Admissions  Status 

Spec.  Admits 

(4/46)9% 

(20/47)43% 

(4/59)7% 

(2/8)25% 

(9/44)21% 

Reg  Admits 

(25/85)29% 

(82/141)58% 

(19/84)23% 

(23/35)66% 

(23/97)24% 

EOP  Status: 

EOP 

(11/78)14% 

(38-77)49% 

3/41)7% 

(5/9)56% 

(5/36)14% 

Non-EOP 

(18-53)34% 

(64/111)58% 

(20/102)20% 

(20/34)59% 

(27/105)26% 

College 

Non-Engineer 

(22/91)24% 

(59/122)48% 

(18/121)15% 

(16/30)53% 

(22/112)20% 

Engineer 

(7/40)18% 

(43/66)65% 

:  23% 

(9/13)69% 

(10/29)34% 

Sex 

Male 

(17/77)22% 

(57/98)58% 

(12/81)15% 

(15/24)62% 

(17/83)20% 

Female 

(12/54)22% 

(45/90)50% 

(11/62)18% 

(10/19)53% 

(15/58)26% 

SAT-Math  Triad' 

200-460 

(8   43)19% 

a  ?<>)3o% 

(6/47)13% 

- 

(3/37)8% 

470-540 

-  15)20% 

(29/54)54% 

(6/35)17% 

(8/17)47". 

(7/32)22% 

550-800 

(11/41)27% 

(46/0)71% 

(10/42)24% 

(16-22)73% 

(18/58)31% 

Note  The  subgroups  are  not  mutually  exclusive  ie  g .  EOP  students  mav  also  have  been  engineering  maiorv 

A  tripartite  division  of  students  by  SAT-Mathematics  scores  was  created  by  dividing  all  students  in  the  study  into  three  groups   of  approximately  equal  (33-33%) 
proportions 
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Persistence  and  Graduation 

Table  2  illustrates  the  persistence  and  graduation  rates  of  MWP  and  non-MWP  UCB  students 
who  entered  the  school  during  the  1978-79  term.  The  proportion  of  MWP  students  who  had 
earned  degrees  or  who  were  still  enrolled  in  a  mathematics-based  major  by  the  spring  1985 
semester  was  65  percent.  The  comparable  rate  among  non-MWP  students  who  entered  the 
university  during  this  period  was  41  percent,  and  the  rate  among  African  American  students 
who  entered  the  university  in  1973  was  39  percent  (both  comparisons  are  significant  at  p<0.001). 
Of  particular  significance  is  the  fact  that  the  proportion  of  MWP  persisters  with  SAT-M  scores 
in  the  lowest  triad  was  higher  than  the  proportion  of  non-MWP  persisters  with  scores  in  the 
highest  triad.  MWP  students  from  the  lowest  triad  group  also  persisted  at  rates  higher  than 
those  of  African  American  students  in  the  historical  control  group  with  SAT-M  scores  in  the  top 
triad. 
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Table  2 
Persistence  and  Graduation*  Among  African  American  Mathematics  lA-Takers,  by  Year 
and  by  MWP  Membership 


Academic  Categories 

Historical  Control 
Group(1973-77) 

MWP  Participants 
(1978-79) 

Non-MWP  Participants  (1978- 
79) 

All  Air.  Amers  Taking 
Math  1A 

N=547  39% 

N=72  65% 

N=209  41% 

Admissions  Status 

Spec.  Admits 

(79/279)27% 

(14/24)58% 

(30/91)33% 

Reg.  Admits 

(113/250)53% 

(33/48)69% 

(55/118)46% 

EOP  Status: 

EOP 

(118/356)33% 

(24/40)60% 

(45/119)38% 

Non-EOP 

(94/191)49% 

(23/32)72% 

(40/90)44% 

College: 

Non-Engineer 

(189/497)38% 

(37/52)71% 

(80/194)41% 

Engineer 

(23/50)46% 

(10/20)50% 

(5/15)33% 

Sex: 

Male 

(76/221)34% 

(23/33)70% 

(38/104)37% 

Female 

(136/326)42% 

(24/39)62% 

(47/105)45% 

SAT-Math  Triad": 

200-460 

(96/251)36% 

(14/24)58% 

(33/100)33% 

470-540 

(46/98)47% 

(10/17)59% 

(13/27)48% 

550-800 

(38/79)48% 

(17/22)77% 

(17/35)49% 

Note:  Ihe  subgroups  are  not  mutually  exclusive  (e.g.,  h(JlJ  students  may  also  have  been  engineering  majors.) 

'  Refer  to  all  students  who  had  either  graduated  or  were  still  enrolled  at  UCB  as  of  the 
fall  1985  semester. 

"  A  tripartite  division  of  students  by  SAT-Mathematics  scores  was  created  by  dividing 
all  students  in  the  study  into  three  groups  of  approximately  equal  (33.33%)  proportions. 
SAT  scores  are  not  available  for  all  students  in  the  sample. 
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Appendix  #14 
BARRON'S  PROFILES  OF  AMERICAN  COLLEGES 

(16TH  EDITION,  1988) 


EXPLANATION: 

The  "College  Admissions  Selector  Ratings"  was  designed  to  give  applicants  to  4  year 
colleges  some  idea  of  the  competition  that  would  be  encountered  when  applying  to  a 
particular  college. 

The  "College  Admissions  Selector"  is  an  index  that  groups  colleges  according  to  degree 
of  admissions  competitiveness. 


"The  Selector  is  not  a  rating  of  colleges  by  academic  standards  or  quality  of  education; 
it  is  rather  attempt  to  describe  ,  in  general  terms  the  situation  a  prospective  student  will 
meet  when  applying  to  college. 


The  criteria  used  to  in  determining  the  category  for  each  college  were: 

median  entrance  examination  scores  for  the  freshman  class  the  SAT  score 
used  was  derived  bye  average  in  the  median  verbal  and  median  math 
scores  The  ACT  score  used  was  the  median  composite  score. 

percentages  of  the  freshmen  scoring  above  500  and  above  600  on  both  the 
math  and  verbal  sections  of  the  SAT. 

percentage  of  freshmen  scoring  above  21  and  above  25  on  the  ACT 

the  percentage  of  freshmen  who  ranked  in  the  upper  fifth  of  their  high 
school  class  and  percentage  who  ranked  in  the  upper  2/5ths 

minimum  grade  point  average  required  for  admission 

percentage  of  applicants  to  freshman  class  who  were  accepted. 
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MOST  COMPETITIVE  =  5 


In  general  these  colleges  require  high  school  rank  in  the  top  10%  to  20%  and  grade 
averages  of  A  to  B+.  Median  freshman  test  scores  at  these  colleges  are  generally  between 
625  and  800  on  the  SAT  and  above  27  on  the  ACT.  In  addition  many  of  these  colleges 
admit  only  a  small  percentage  of  those  who  apply— usually  fewer  than  one- third. 


Brown 

Cornell 

Harvard 

Tufts 

Michigan 


William  &  Mary 

Duke 

MIT 

U  of  Perm 


-  Columbia 

-  Georgetown 

-  John  Hopkins 

-  Notre  Dame 


HIGHLY  COMPETITIVE  =  4 

Colleges  in  this  group  look  for  students  with  grade  averages  of  B+  to  B  and  accept  most 
of  their  students  from  the  top  20%  to  35%  of  the  freshman  class.  Median  freshman  test 
scores  at  these  colleges  range  from  575  to  625  on  the  SAT  and  from  26  to  27  on  the  ACT. 
These  schools  generally  accept  between  one-third  and  one-half  of  their  applicants. 


Boston  College 

-  Boston  University 

-  Barnard 

Columbia  Gen. 

Studies 

-  Emory 

-  Oberlin 

Rutgers 

-  Smith 

-  Tulane 

UVA 

-  Berkeley 

-  Fairfield 

SUNY  Albany 

-  U  of  Chicago 

-  Union 

UCLA 

-  Vanderbilt 

-  Villanova 

Wake  Forest 

-  U  of  Illinois 

-UNC 

VERY  COMPETITIVE  =  3 

The  colleges  in  this  category  admit  students  whose  averages  are  no  less  than  B-  and  who 

rank  in  the    top  35%  to  50%  of  their  graduating  class.  They  report  a  median  freshman 

test  scores  in  the  535  to  575  range  on  the  sat  and  from  23  to  25  on  the  act. 

The  schools  in  this  category  generally  accept  between  one-half  and  three  quarters  of  their 

applicants. 


SUNY 

-  Texas  A&M 

-  Baylor 

Cal  Schools 

-  U  of  Mass 

-  Miami 

Providence 

-  Syracuse 

-use 

Adelphi 

-CCNY 

-  Creighton 

Florida  International 

-  Wheaton 

-GWU 

Michigan  State 
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COMPETITIVE  =  2 

This  category  is  a  very  broad  one,  covering  colleges  that  generally  have  median 
freshman  test  scores  between  450  and  525  on  the  SAT  and  between  19  and  22  on  the 
ACT.  Some  of  these  colleges  require  that  students  have  high  school  averages  of  B-  or 
better,  although  others  state  a  minimum  of  C+  or  C.  Generally  these  colleges  prefer 
students  in  the  top  50%  to  65%  of  the  graduating  high  school  class  and  accept  between 
75%  and  85%  of  their  applicants. 


Cal  State 

-  Hunter 

-  Indiana 

Minnesota 

-  Rutgers  Newark 

-  U  Cal  Sac 

U  of  Alabama 

-  SUNY  Buffalo 

-  Howard 

Virginia  State 

-  St  Marys  TX 

-  Spelman 

Davis 

-  U  of  Delaware 

LESS  COMPETITIVE  =  1 

Included  in  this  category  are  colleges  with  median  freshman  test  scores  below  450  on  the 
SAT  and  below  19  on  the  ACT;  some  colleges  that  require  entrance  examinations  but  do 
not  report  median  scores;  and  colleges  that  admit  students  with  averages  below  C  who 
rank  in  the  top  65%  of  their  graduating  class.  These  colleges  usually  admit  85%  percent 
of  their  applicants. 


Bennet  College 

-  Bethune  Cookman 

-Fisk 

Florida  A&M 

-  Delaware  State 

-  Hampton 

Lincoln 

-  Morehouse 

-  Talladega 

Tougaloo 

-  Md.  Eastern  Shore 

-UDC 

Morgan  State 

-  Xavier  Ohio 

-  Oakland 

NONCOMPETITIVE  =  0 


The  colleges  in  this  category  generally  only  require  evidence  of  graduation  from  an 
accredited  high  school  (although  they  may  also  require  the  completion  of  a  certain 
number  of  high  school  units). 

Some  require  that  entrance  examinations  be  taken  for  placement  purposes  only  or  only 
by  graduates  of  unaccredited  high  schools  or  only  by  out-of-state  students.  In  some 
cases,  insufficient  capacity  may  compel  a  college  in  this  category  to  limit  the  number  of 
students  that  are  accepted  generally,  however,  if  a  collage  accepts  all  its  applicants,  it 
automatically  falls  in  this  category.  Colleges  are  rated  Noncompetitive  if  they  admit  all 
state  residents,  but  have  some  requirements  for  nonresidents. 


Medgar  Evers 


Grambling 


-  Southern 
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Appendix  #15 
Diagnostic  and  Admissions  Tests 


This  appendix  represents  a  description  of  diagnostic  and  admissions  tests  currently 
recommended  by  the  Division  of  Disadvantaged  Assistance  in  its  1994  Technical 
Assistance  Manual  to  HCOP  grantee  applicants. 


DIAGNOSTIC  TESTS 


COOPERATIVE  MATHEMATICS  TEST 

Measures  achievement  in  major  mathematical  content  areas  ranging  from  arithmetic  to 
calculus. 


IOWA  SILENT  READING  TEST 

To  assess  student's  reading  ability  and  comprehension.  Grantees  use  levels  II  and  III  to 
plan  preliminary  education  and  retention  programs. 


IOWA  TESTS  OF  EDUCATIONAL  DEVELOPMENT 

A  high  school  achievement  test  battery  to  assess  intellectual  skills  that  are  important  in 
adult  life  and  provide  the  basis  for  continued  learning.  May  be  useful  to  grantees 
planning  preliminary  educational  programs  for  freshman  and  sophomore 
undergraduates. 


MYERS  BRIGGS  TYPE  INDICATOR 
To  assess  individual  personality  dimensions. 

NELSON-DENNY  READING  TEST 

To  assess  student  achievement  and  progress  in  vocabulary,  comprehension,  and  reading 
rates.  Used  by  grantees  to  plan  preliminary  education  and  retention  programs.  Other 
forms  of  the  test  available  for  use  as  pre-  and  post-tests. 
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STANFORD  DIAGNOSTIC  READING  TEST 

To  assess  the  strengths  and  weaknesses  of  students'  reading  ability.   Grantees  use  the 
4th  level  to  plan  preliminary  education  and  retention  programs. 


STRONG  INTEREST  INVENTORY 

To  compare  an  individual's  interests  with  the  interest  of  individuals  already  in  a  variety 
of  occupations. 


TEST  ANXIETY  PROFILE 
To  identify  the  types  of  academic  situations  that  produce  anxiety  in  the  student. 

WATSON-GLASER  CRITICAL  THINKING  APPRAISAL 

To  assess  critical  thinking  abilities.   Evaluation  of  gifted  and  talented  individuals. 

ADMISSIONS  TESTS 

MEDICAL  COLLEGE  ADMISSION  TEST  (MCAT) 


To  assess  understanding  of  science  concepts  and  principles  identified  as  prerequisite  to 
the  student  and  practice  of  medicine. 

To  evaluate  basic  analytical  skills  in  the  context  of  medically  relevant  problems  and  data. 

To  help  admissions  committees  predict  which  of  their  applicants  will  perform  adequately 
in  the  medical  school  curriculum. 


DENTAL  ADMISSION  TEST  (DAT) 

To  evaluate  academic  background  and  motor  abilities  required  for  dental  education  and 
practice. 
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VETERINARY  COLLEGE  ADMISSION  TEST  (VCAT) 
To  test  academic  achievement  in  areas  necessary  for  the  study  of  veterinary  medicine. 

OPTOMETRY  ADMISSION  TEST  (OAT) 

To  measure  general  academic  ability  and  comprehension  of  scientific  information  for 
admission  to  colleges  of  Optometry. 

PHARMACY  COLLEGE  ADMISSION  TEST  (PCAT) 

Assesses  general  levels  of  academic  knowledge.    One  criterion  for  admission  to  most 
colleges  of  pharmacy. 

GRADUATE  RECORD  EXAMINATION  (GRE) 
To  qualify  for  admission  into  a  graduate  program. 

ALLIED  HEALTH  PROFESSIONS  ADMISSION  TEST  (AHPAT) 
To  qualify  for  admission  to  advanced  level  allied  health  studies. 

DENTAL  HYGIENE  COLLEGE  ADMISSION  TEST  (DHCAT) 
To  test  academic  achievement  in  areas  necessary  for  the  study  of  dental  hygiene. 


123 


